Eye center of rotation determination with one or more eye tracking cameras

ABSTRACT

A display system can include a head-mounted display configured to project light to an eye of a user to display virtual image content at different amounts of divergence and collimation. The display system can include an inward-facing imaging system possibly comprising a plurality of cameras that image the user&#39;s eye and glints for thereon and processing electronics that are in communication with the inward-facing imaging system and that are configured to obtain an estimate of a center of rotation of the user&#39;s eye using cornea data derived from the glint images. The display system may render virtual image content with a render camera positioned at the determined position of the center of rotation of said eye.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/874,867 filed on Jul. 16, 2019, which is titled “EYE CENTER OFROTATION DETERMINATION WITH ONE OR MORE EYE TRACKING CAMERAS,” thecontents of which is herein incorporated by reference in its entirety.This application is related to U.S. application Ser. No. 16/250,931,which is titled “EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANESELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS,” and wasfiled on Jan. 17, 2019, and U.S. Patent Pub. 2018/0018515, which istitled “IRIS BOUNDARY ESTIMATION USING CORNEA CURVATURE” and waspublished on Jan. 18, 2018, which are hereby incorporated by referencein their entirety.

FIELD

The present disclosure relates to display systems, virtual reality, andaugmented reality imaging and visualization systems and, moreparticularly, to eye tracking using a center of rotation of an eyecalculated using cornea data.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality”, “augmentedreality”, or “mixed reality” experiences, wherein digitally reproducedimages or portions thereof are presented to a user in a manner whereinthey seem to be, or may be perceived as, real. A virtual reality, or“VR”, scenario typically involves presentation of digital or virtualimage information without transparency to other actual real-world visualinput; an augmented reality, or “AR”, scenario typically involvespresentation of digital or virtual image information as an augmentationto visualization of the actual world around the user; a mixed reality,or “MR”, related to merging real and virtual worlds to produce newenvironments where physical and virtual objects co-exist and interact inreal time. As it turns out, the human visual perception system is verycomplex, and producing a VR, AR, or MR technology that facilitates acomfortable, natural-feeling, rich presentation of virtual imageelements amongst other virtual or real-world imagery elements ischallenging. Systems and methods disclosed herein address variouschallenges related to VR, AR and MR technology.

SUMMARY

Various examples of depth plane selection in a mixed reality system aredisclosed.

A display system can be configured to project light to an eye of a userto display virtual image content in a vision field of said user. Theuser's eye may have a cornea, an iris, a pupil, a lens, a retina, and anoptical axis extending through said lens, pupil, and cornea. The displaysystem can include a frame configured to be supported on a head of theuser, a head-mounted display disposed on the frame, one or more eyetracking cameras configured to image the user's eye, and processingelectronics in communication with the display and the one or more eyetracking cameras, the processing electronics configured to obtain anestimate of a parameter of the eye based on images of said eye obtainedwith said one or more eye tracking cameras. In some implementations, theparameter of the eye comprises a center of curvature of the cornea(e.g., the center of curvature as measured at the corneal apex). In someimplementations, the center of curvature of the cornea or the center ofthe cornea refers to the center of curvature of a portion of the corneaor the center of curvature of a spherical surface that coincides with aportion of the surface of the cornea. For example, in someimplementations, the center of curvature of the cornea or the center ofthe cornea refers to the center of curvature of the cornea apex or thecenter of curvature of a spherical surface that coincides with a portionof the surface of the corneal apex. In some implementations, theparameter of the eye comprises the center of rotation of said eye. Otherparameters and information may be determined as well.

In some implementations, the display is configured to project light intosaid user's eye to display virtual image content to the user's visionfield at different amounts of at least one of divergence and collimationand thus the displayed virtual image content appears to originate fromdifferent depths. In some implementations, the displayed virtual imagecontent appears to originate from different depths at different periodsof time.

Various examples of display systems that project light to one or moreeyes of a user to display virtual image content in a vision field ofsaid user are described herein such as the examples enumerated below:

Part I

Example 1: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising: a frame configured to be supported on ahead of the user; a head-mounted display disposed on the frame, saiddisplay configured to project light into said user's eye to displayvirtual image content to the user's vision field; first and second eyetracking cameras configured to image the user's eye; a plurality oflight emitters; and processing electronics in communication with thedisplay and the first and second eye tracking cameras, the processingelectronics configured to: receive images of the user's eye captured bythe first and second eye tracking cameras, glint reflections of thedifferent light emitters observable in said images of the eye capturedby the first and second eye tracking cameras; and estimate a location ofsaid center of corneal curvature of the user's eye based on the locationof the glint reflections in said images produced by both said first andsecond eye tracking camera and based on the location of both the firstand second eye tracking cameras and the locations of the emitters thatproduced said respective glint reflections.

Example 2: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising: a frame configured to be supported on ahead of the user; a head-mounted display disposed on the frame, saiddisplay configured to project light into said user's eye to displayvirtual image content to the user's vision field; first and second eyetracking cameras configured to image the user's eye; a plurality oflight emitters; and processing electronics in communication with thedisplay and the first and second eye tracking cameras, the processingelectronics configured to: receive images of the user's eye captured bythe first and second eye tracking cameras, glint reflections of thedifferent light emitters observable in said images of the eye capturedby the first and second eye tracking cameras; and estimate a location ofsaid center of rotation of the user's eye based on the location of theglint reflections in said images produced by both said first and secondeye tracking camera and based on said the location of both the first andsecond eye tracking cameras and the locations of the emitters thatproduced said glint reflections for multiple eye poses.

Example 3: A method of determining one or more parameters associatedwith an eye for rendering virtual image content in a display systemconfigured to project light to an eye of a user to display the virtualimage content in a vision field of said user, said eye having a cornea,said method comprising: with a plurality of eye tracking camerasconfigured to image the eye of the user and a plurality of lightemitters disposed with respect to said eye to form glints thereon,capturing a plurality of images of the eye of the user, said imagescomprising a plurality of glints; and obtaining an estimate of a centerof rotation of said eye based on the plurality of glints, whereinobtaining an estimate of the center of rotation of said eye comprises:determining a plurality of estimates of the center of corneal curvatureof the user's eye based on the plurality of glints; generating athree-dimensional surface from the plurality of estimates of the centerof the corneal curvature; and determining the estimate of the center ofrotation of the user's eye using the three-dimensional surface.

Example 4: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising: a frame configured to be supported on ahead of the user; a head-mounted display disposed on the frame, saiddisplay configured to project light into said user's eye to displayvirtual image content; first and second eye tracking cameras configuredto image the user's eye; and processing electronics in communicationwith the display and the first and second eye tracking cameras, theprocessing electronics configured to: receive multiple pairs of capturedimages of the user's eye from the first and second eye tracking cameras;for pairs of images received from the first and second eye trackingcameras, respectively, obtain an estimate of a center of cornealcurvature of the user's eye based at least in part on the respectivepair of captured images; determine a three-dimensional surface based onthe estimated centers of corneal curvature of the user's eye obtainedbased on the multiple pairs of captured images of the user's eyereceived from the respective first and second eye tracking cameras; andidentify a center of curvature of the 3D surface to obtain an estimateof a center of rotation of the user's eye.

Example 5: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising: a frame configured to be supported on ahead of the user; a head-mounted display disposed on the frame, saiddisplay configured to project light into said user's eye to displayvirtual image content to the user's vision field; an eye tracking cameraconfigured to image the user's eye; a plurality of light emitters; andprocessing electronics in communication with the display and the eyetracking camera, the processing electronics configured to: receiveimages of the user's eye captured by the eye tracking camera at a firstand second location, glint reflections of the different light emittersobservable in said images of the eye captured by the eye trackingcamera; and estimate a location of said center of corneal curvature ofthe user's eye based on the location of the glint reflections in saidimages produced by said eye tracking camera and based on the location ofthe eye tracking camera and the locations of the emitters that producedsaid respective glint reflections.

Example 6: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising: a frame configured to be supported on ahead of the user; a head-mounted display disposed on the frame, saiddisplay configured to project light into said user's eye to displayvirtual image content to the user's vision field; an eye tracking cameraconfigured to image the user's eye; a plurality of light emitters; andprocessing electronics in communication with the display and the eyetracking camera, the processing electronics configured to: receiveimages of the user's eye captured by the eye tracking camera at a firstcamera and second location, glint reflections of the different lightemitters observable in said images of the eye captured by the eyetracking camera; and estimate a location of said center of rotation ofthe user's eye based on the location of the glint reflections in saidimages produced by said eye tracking camera and based on said first andsecond location of the eye tracking camera and the locations of theemitters that produced said glint reflections for multiple eye poses.

Example 7: A method of determining one or more parameters associatedwith an eye for rendering virtual image content in a display systemconfigured to project light to an eye of a user to display the virtualimage content in a vision field of said user, said eye having a cornea,said method comprising: with an eye tracking camera configured to imagethe eye of the user and a plurality of light emitters disposed withrespect to said eye to form glints thereon, capturing a plurality ofimages of the eye of the user, said images comprising a plurality ofglints; and obtaining an estimate of a center of rotation of said eyebased on the plurality of glints, wherein obtaining an estimate of thecenter of rotation of said eye comprises: determining a plurality ofestimates of the center of corneal curvature of the user's eye based onthe plurality of glints; generating a three-dimensional surface from theplurality of estimates of the center of the corneal curvature; anddetermining the estimate of the center of rotation of the user's eyeusing the three-dimensional surface.

Example 8: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising: a frame configured to be supported on ahead of the user; a head-mounted display disposed on the frame, saiddisplay configured to project light into said user's eye to displayvirtual image content; an eye tracking camera configured to image theuser's eye; and processing electronics in communication with the displayand the eye tracking camera, the processing electronics configured to:receive multiple pairs of captured images of the user's eye from the eyetracking camera; for pairs of images received from the eye trackingcamera, respectively, obtain an estimate of a center of cornealcurvature of the user's eye based at least in part on the respectivepair of captured images; determine a three-dimensional surface based onthe estimated centers of corneal curvature of the user's eye obtainedbased on the multiple pairs of captured images of the user's eyereceived from the eye tracking camera; and identify a center ofcurvature of the 3D surface to obtain an estimate of a center ofrotation of the user's eye.

Example 9: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising: a frame configured to be supported on ahead of the user; a head-mounted display disposed on the frame, saiddisplay configured to project light into said user's eye to displayvirtual image content to the user's vision field; at least one eyetracking camera configured to image the user's eye; a plurality of lightemitters; and processing electronics in communication with the displayand the eye tracking camera, the processing electronics configured to:receive images of the user's eye captured by the at least one eyetracking camera at a first and second location, glint reflections of thedifferent light emitters observable in said images of the eye capturedby the eye tracking camera; and estimate a location of said center ofcorneal curvature of the user's eye based on the location of the glintreflections in said images produced by said at least one eye trackingcamera and based on the location of the at least one eye tracking cameraand the locations of the emitters that produced said respective glintreflections.

Part II

Example 1: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising:

a frame configured to be supported on a head of the user;

a head-mounted display disposed on the frame, said display configured toproject light into said user's eye to display virtual image content tothe user's vision field;

first and second eye tracking cameras configured to image the user'seye;

a plurality of light emitters; and

processing electronics in communication with the display and the firstand second eye tracking cameras, the processing electronics configuredto:

-   -   receive images of the user's eye captured by the first and        second eye tracking cameras, glint reflections of the different        light emitters observable in said images of the eye captured by        the first and second eye tracking cameras; and    -   estimate a location of a center of corneal curvature of the        user's eye based on the location of the glint reflections in        said images produced by both said first and second eye tracking        camera and based on the location of both the first and second        eye tracking cameras and the locations of the emitters that        produced said respective glint reflections.

Example 2: The display system of Example 1, wherein said processingelectronics is configured to:

-   -   based on the location of the glint reflections in one or more        images produced by said first eye tracking camera and based on        the location of the first eye tracking camera and the location        of the emitters that produced said glint reflections, determine        a first direction toward the center of corneal curvature of the        user's eye; and    -   based on the location of the glint reflections in one or more        images produced by said second eye tracking camera and based on        the location of the second eye tracking camera and the location        of the emitters that produced said glint reflections, determine        a second direction toward the center of corneal curvature of the        user's eye.

Example 3: The display system of Example 2, wherein said processingelectronics is configured to determine the first direction by:

-   -   defining a first plane that includes the first eye tracking        camera, a location of a first glint reflection and a location of        the light emitter corresponding to said first glint reflection;    -   defining a second plane that includes the first eye tracking        camera, a location of a second glint reflection and a location        of the light emitter corresponding to said second glint        reflection; and    -   determining a region of convergence of the first plane and the        second plane, the region of convergence extending along the        first direction.

Example 4: The display system of Example 3, said processing electronicsare configured to determine the second direction by:

-   -   defining a third plane that includes the second eye tracking        camera, the location of a third glint reflection, and a location        of the light emitter corresponding to said third glint        reflection;    -   defining a fourth plane that includes the second eye tracking        camera, the location of a fourth glint reflection, and a        location of the light emitter corresponding to said fourth glint        reflection; and    -   determining a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        second direction.

Example 5: The display system of any of the Examples above, wherein saidprocessing electronics is configured to estimate a location of saidcenter of corneal curvature of the user's eye based on said first andsecond directions toward the center of the corneal curvature of theuser's eye.

Example 6: The display system of any of the Examples above, wherein saidprocessing electronics is configured to:

-   -   determine said first direction along which the center of corneal        curvature of the user's eye is estimated to be located based on        at least one first image received from the first eye tracking        camera; and    -   determine said second direction along which the center of        corneal curvature of the user's eye is estimated to be located        based on at least one second image received from the second eye        tracking camera, said first and second directions converging        toward a region.

Example 7: The display system of any of the Examples above, wherein saidprocessing electronics is configured to:

-   -   obtain an estimate of a center of corneal curvature of the        user's eye based on the convergence of the first and second        directions.

Example 8: The display system of any of the Examples above, wherein saidprocessing electronics is configured to estimate a location of saidcenter of corneal curvature of the user's eye by identifying a region ofconvergence of said first and second directions toward the center of thecorneal curvature of the user's eye.

Example 9: The display system of any of the Examples above, wherein saidprocessing electronics is configured to obtain an estimate of a centerof rotation of the user's eye based on multiple determinations of thecenter of corneal curvature of the user's eye for different eye poses.

Example 10: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a locus of pointscorresponding to estimates of the center of corneal curvature of theuser's eye for different eye poses.

Example 11: The display system of Example 10, wherein said processingelectronics is configured to obtain an estimate of a center of rotationof the user's eye based on said locus of points corresponding toestimates of the center of corneal curvature of the user's eye fordifferent eye poses.

Example 12: The display system of Examples 10 or 11, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye.

Example 13: The display system of Examples 10 or 11, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye by estimating a center of curvature of said surface.

Example 14: The display system of Examples 10 or 11, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye by determining a region where a plurality of normalsto said surface converge.

Example 15: The display system of any of Examples 12, 13, or 14, whereinsaid processing electronics is configured to fit said surface to saidlocus of points to obtain said surface.

Example 16: The display system of any of the Examples above, whereinsaid processing electronics is configured to use a render camera torender virtual images to be presented to the eye of the user, saidrender camera having a position determined by said center of rotation.

Example 17: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content to the user's vision field at differentamounts of at least one of divergence and collimation and thus thedisplayed virtual image content appears to originate from differentdepths.

Example 18: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising:

-   -   a frame configured to be supported on a head of the user;    -   a head-mounted display disposed on the frame, said display        configured to project light into said user's eye to display        virtual image content to the user's vision field;    -   first and second eye tracking cameras configured to image the        user's eye;    -   a plurality of light emitters; and    -   processing electronics in communication with the display and the        first and second eye tracking cameras, the processing        electronics configured to:        -   receive images of the user's eye captured by the first and            second eye tracking cameras, glint reflections of the            different light emitters observable in said images of the            eye captured by the first and second eye tracking cameras;            and        -   estimate a location of said center of rotation of the user's            eye based on the location of the glint reflections in said            images produced by both said first and second eye tracking            camera and based on said the location of both the first and            second eye tracking cameras and the locations of the            emitters that produced said glint reflections for multiple            eye poses.

Example 19: The display system of Example 18, wherein to obtain anestimate of the center of rotation of said eye, the processingelectronics are configured to:

-   -   determine a plurality of estimates of a center of corneal        curvature of the user's eye based a plurality of glint        reflections for multiple eye poses; and    -   determine the estimate of the center of rotation of the user's        eye based on the plurality of estimates of the center of corneal        curvature of the user's eye for said multiple eye poses.

Example 20: The display system of Example 19, wherein to determine saidplurality of estimates of the corneal curvature of the user's eye, theprocessing electronics are configured to:

-   -   determine a first direction toward the center of corneal        curvature based on the respective locations of at least a        portion of said plurality of emitters and a first camera of the        eye tracking cameras;    -   determine a second direction toward the center of corneal        curvature based on the respective locations of at least a        portion of said plurality of emitters and a second camera of the        eye tracking cameras; and    -   determine an estimate of the center of corneal curvature of the        user's eye based on said the first and second directions.

Example 21: The display system of Example 20, wherein said processingelectronics is configured to determine the first direction by:

-   -   defining a first plane that includes the first eye tracking        camera, a location of a first glint reflection and a location of        the light emitter corresponding to said first glint reflection;    -   defining a second plane that includes the first eye tracking        camera, a location of a second glint reflection and a location        of the light emitter corresponding to said second glint        reflection; and    -   determining a region of convergence of the first plane and the        second plane, the region of convergence extending along the        first direction.

Example 22: The display system of Example 21, said processingelectronics are configured to determine the second direction by:

-   -   defining a third plane that includes the second eye tracking        camera, the location of a third glint reflection, and a location        of the light emitter corresponding to said third glint        reflection;    -   defining a fourth plane that includes the second eye tracking        camera, the location of a fourth glint reflection, and a        location of the light emitter corresponding to said fourth glint        reflection; and    -   determining a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        second direction.

Example 23: The display system of any of Examples 20-22, wherein todetermine said plurality of estimates of the corneal curvature of theuser's eye, the processing electronics are configured to:

-   -   determine a region of convergence between the first direction        and second direction to determine an estimate of the center of        corneal curvature of the user's eye.

Example 24: The display system of any of Examples 19-23, wherein toobtain an estimate of the center of rotation of said eye, the processingelectronics are configured to:

-   -   generate a three-dimensional surface associated with the        plurality of estimates of the center of the corneal curvature;        and    -   determine the estimate of the center of rotation of the user's        eye based on the three-dimensional surface.

Example 25: The display system of Example 24, wherein to generate athree-dimensional surface associated with the plurality of estimates ofthe center of the corneal curvature, the processing electronics areconfigured to fit a surface to the plurality of estimates of the centerof the corneal curvature.

Example 26: The display system of Example 24, wherein to generate athree-dimensional surface associated with the plurality of estimates ofthe center of the corneal curvature, the processing electronics areconfigured to fit a spherical surface to the plurality of estimates ofthe center of the corneal curvature.

Example 27: The display system of any of Example 24-26, wherein todetermine the estimate of the center of rotation of the user's eye, theprocessing electronics are configured to:

-   -   determine two or more normals to the three-dimensional surface;        and    -   determine a region of convergence of the two or more normals,        wherein the region of convergence comprises the estimate of the        center of rotation of the user's eye.

Example 28: The display system of any of Examples 21-27, wherein the oneor more images of the user's eye comprise one or more images associatedwith different gaze vectors of the user's eye.

Example 29: The display system of any of Examples 21-28, wherein theprocessing electronics are configured to map the cornea of the user'seye using a gaze target.

Example 30: The display system of any of Examples 18-29, wherein saidprocessing electronics is configured to use a render camera to rendervirtual images to be presented to the eye of the user, said rendercamera having a position determined by said center of rotation.

Example 31: The display system of any of Examples 18-30, wherein saiddisplay is configured to project light into said user's eye to displayvirtual image content to the user's vision field at different amounts ofat least one of divergence and collimation and thus the displayedvirtual image content appears to originate from different depths.

Example 32: A method of determining one or more parameters associatedwith an eye for rendering virtual image content in a display systemconfigured to project light to an eye of a user to display the virtualimage content in a vision field of said user, said eye having a cornea,said cornea having a center of curvature, said method comprising:

-   -   with a plurality of eye tracking cameras configured to image the        eye of the user and a plurality of light emitters disposed with        respect to said eye to form glints thereon, capturing a        plurality of images of the eye of the user, said images        comprising a plurality of glints; and    -   obtaining an estimate of a center of rotation of said eye based        on the plurality of glints, wherein obtaining an estimate of the        center of rotation of said eye comprises:        -   determining a plurality of estimates of the center of            corneal curvature of the user's eye based on the plurality            of glints;        -   generating a three-dimensional surface from the plurality of            estimates of the center of the corneal curvature; and        -   determining the estimate of the center of rotation of the            user's eye using the three-dimensional surface.

Example 33: The method of Example 32, wherein determining the pluralityof estimates of the corneal curvature of the user's eye comprises:

-   -   determining a first vector directed toward the center of corneal        curvature based on the locations of at least a portion of the        plurality of light emitters and the location of a first camera        of the plurality of eye tracking cameras;    -   determining a second vector directed toward the center of        corneal curvature based on locations of at least a portion of        the plurality of light emitters and the location of a second        camera of the plurality of eye tracking cameras; and    -   determining a region of convergence between the first vector and        second vector to determine an estimate of the center of corneal        curvature of the user's eye.

Example 34: The method of Example 33, wherein the first direction isdetermined by:

-   -   defining a first plane that includes the first eye tracking        camera, a location of a first glint reflection and a location of        the light emitter corresponding to said first glint reflection,    -   defining a second plane that includes the first eye tracking        camera, a location of a second glint reflection and a location        of the light emitter corresponding to said second glint        reflection; and    -   determining a region of convergence of the first plane and the        second plane, the region of convergence extending along the        first direction.

Example 35: The method of Example 33, wherein the second direction isdetermined by:

-   -   defining a third plane that includes the second eye tracking        camera, the location of a third glint reflection, and a location        of the light emitter corresponding to said third glint        reflection;    -   defining a fourth plane that includes the second eye tracking        camera, the location of a fourth glint reflection, and a        location of the light emitter corresponding to said fourth glint        reflection; and    -   determining a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        second direction.

Example 36: The method of any of Examples 32-35, wherein generating athree-dimensional surface from the plurality of estimates of the centerof the corneal curvature comprises fitting a surface to the plurality ofestimates of the center of the corneal curvature.

Example 37: The method of any of Examples 32-35, wherein generating athree-dimensional surface from the plurality of estimates of the centerof the corneal curvature comprises fitting a sphere to the plurality ofestimates of the center of the corneal curvature.

Example 38: The method of any of Examples 32-37, wherein determining theestimate of the center of rotation of the user's eye comprises:

-   -   determining two or more vectors normal to the three-dimensional        surface; and    -   determining a region of convergence of the two or more vectors        normal to the three-dimensional surface, wherein the region of        convergence comprises the estimate of the center of rotation of        the user's eye.

Example 39: The method of any of Examples 32-38, wherein the pluralityof images of the user's eye comprise images associated with differentgaze directions of the user's eye.

Example 40: The method of any of Examples 32-39, further comprisingmapping the cornea of the user's eye using a gaze target.

Example 41: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising:

-   -   a frame configured to be supported on a head of the user;    -   a head-mounted display disposed on the frame, said display        configured to project light into said user's eye to display        virtual image content;    -   first and second eye tracking cameras configured to image the        user's eye; and    -   processing electronics in communication with the display and the        first and second eye tracking cameras, the processing        electronics configured to:        -   receive multiple pairs of captured images of the user's eye            from the first and second eye tracking cameras;        -   for pairs of images received from the first and second eye            tracking cameras, respectively, obtain an estimate of a            center of corneal curvature of the user's eye based at least            in part on the respective pair of captured images;        -   determine a three-dimensional surface based on the estimated            centers of corneal curvature of the user's eye obtained            based on the multiple pairs of captured images of the user's            eye received from the respective first and second eye            tracking cameras; and        -   identify a center of curvature of the 3D surface to obtain            an estimate of a center of rotation of the user's eye.

Example 42: The display system of Example 41, wherein said processingelectronics is configured to fit a three-dimensional surface to theestimated centers of corneal curvature of the user's eye obtained basedon the multiple pairs of captured images of the user's eye received fromthe respective first and second eye tracking cameras.

Example 43: The display system of Examples 41 or 42, wherein to obtainthe estimate of the center of corneal curvature of the user's eye basedat least in part on the respective pair of captured images, theprocessing electronics are configured to:

-   -   determine a first vector along which the center of corneal        curvature of the user's eye is estimated to be located based on        a first image received from the first eye tracking camera;    -   determine a second vector along which the center of corneal        curvature of the user's eye is estimated to be located based on        a second image received from the second eye tracking camera, the        first and second images corresponding to one of said pairs of        images; and    -   identify a region of convergence between paths extending in the        direction of the first vector and the second vector to obtain an        estimate of a center of corneal curvature of the user's eye.

Example 44: The display system of Example 43, further comprising:

-   -   a plurality of light emitters configured to illuminate the        user's eye to form glint reflections thereon,    -   wherein to determine the first vector based on the first image        of the pair of captured images, the processing electronics are        configured to:        -   define a first plane that includes the first eye tracking            camera, a location of a first glint reflection and a            location of the light emitter corresponding to said first            glint reflection;        -   define a second plane that includes the first eye tracking            camera, a location of a second glint reflection and a            location of the light emitter corresponding to said second            glint reflection; and        -   identify a region of convergence of the first plane and the            second plane, the region of convergence extending along the            direction of the first vector.

Example 45: The display system of Example 44, wherein to determine thesecond vector based on the second image in each pair of captured images,the processing electronics are configured to:

-   -   define a third plane that includes the second eye tracking        camera, the location of a third glint reflection, and a location        of the light emitter corresponding to said third glint        reflection;    -   define a fourth plane that includes the second eye tracking        camera, the location of a fourth glint reflection, and a        location of the light emitter corresponding to said fourth glint        reflection; and    -   determine a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        direction of the second vector.

Example 46: The display system of any of Examples 41-45, wherein saidprocessing electronics is configured to use a render camera to rendervirtual images to be presented to the eye of the user, said rendercamera having a position determined by said center of rotation.

Example 47: The display system of any of Examples 41-46, wherein saiddisplay is configured to project light into said user's eye to displayvirtual image content to the user's vision field at different amounts ofat least one of divergence and collimation and thus the displayedvirtual image content appears to originate from different depths.

Example 48: The display system of any of the Examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 49: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising:

-   -   a frame configured to be supported on a head of the user;    -   a head-mounted display disposed on the frame, said display        configured to project light into said user's eye to display        virtual image content to the user's vision field;    -   an eye tracking camera configured to image the user's eye;    -   a plurality of light emitters; and    -   processing electronics in communication with the display and the        eye tracking camera, the processing electronics configured to:        -   receive images of the user's eye captured by the eye            tracking camera, glint reflections of the different light            emitters observable in said images of the eye captured by            the eye tracking camera; and        -   estimate a location of a center of corneal curvature of the            user's eye based on the location of the glint reflections in            said images produced by said eye tracking camera and based            on the location of the eye tracking camera and the locations            of the emitters that produced said respective glint            reflections.

Example 50: The display system of Example 49, wherein said processingelectronics is configured to:

-   -   based on the location of the glint reflections in one or more        images produced by said eye tracking camera and based on the        location of the eye tracking camera and the location of the        emitters that produced said glint reflections, determine a first        direction toward the center of corneal curvature of the user's        eye; and    -   based on the location of the glint reflections in one or more        images produced by said eye tracking camera and based on the        location of the eye tracking camera and the location of the        emitters that produced said glint reflections, determine a        second direction toward the center of corneal curvature of the        user's eye.

Example 51: The display system of Example 50, wherein said processingelectronics is configured to determine the first direction by:

-   -   defining a first plane that includes the location of the eye        tracking camera, a location of a first glint reflection and a        location of the light emitter corresponding to said first glint        reflection;    -   defining a second plane that includes the location of the eye        tracking camera, a location of a second glint reflection and a        location of the light emitter corresponding to said second glint        reflection; and    -   determining a region of convergence of the first plane and the        second plane, the region of convergence extending along the        first direction.

Example 52: The display system of Example 51, said processingelectronics are configured to determine the second direction by:

-   -   defining a third plane that includes the location of the eye        tracking camera, the location of a third glint reflection, and a        location of the light emitter corresponding to said third glint        reflection;    -   defining a fourth plane that includes the location of the eye        tracking camera, the location of a fourth glint reflection, and        a location of the light emitter corresponding to said fourth        glint reflection; and    -   determining a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        second direction.

Example 53: The display system of any of the Examples above, whereinsaid processing electronics is configured to estimate a location of saidcenter of corneal curvature of the user's eye based on said first andsecond directions toward the center of the corneal curvature of theuser's eye.

Example 54: The display system of any of the Examples above, whereinsaid processing electronics is configured to:

-   -   determine said first direction along which the center of corneal        curvature of the user's eye is estimated to be located based on        at least one first image received from the location of the eye        tracking camera; and    -   determine said second direction along which the center of        corneal curvature of the user's eye is estimated to be located        based on at least one second image received from the location of        the eye tracking camera, said first and second directions        converging toward a region.

Example 55: The display system of any of the Examples above, whereinsaid processing electronics is configured to:

-   -   obtain an estimate of a center of corneal curvature of the        user's eye based on the convergence of the first and second        directions.

Example 56: The display system of any of the Examples above, whereinsaid processing electronics is configured to estimate a location of saidcenter of corneal curvature of the user's eye by identifying a region ofconvergence of said first and second directions toward the center of thecorneal curvature of the user's eye.

Example 57: The display system of any of the Examples above, whereinsaid processing electronics is configured to obtain an estimate of acenter of rotation of the user's eye based on multiple determinations ofthe center of corneal curvature of the user's eye for different eyeposes.

Example 58: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a locus of pointscorresponding to estimates of the center of corneal curvature of theuser's eye for different eye poses.

Example 59: The display system of Example 58, wherein said processingelectronics is configured to obtain an estimate of a center of rotationof the user's eye based on said locus of points corresponding toestimates of the center of corneal curvature of the user's eye fordifferent eye poses.

Example 60: The display system of Examples 58 or 59, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye.

Example 61: The display system of Examples 58 or 59, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye by estimating a center of curvature of said surface.

Example 62: The display system of Examples 58 or 59, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye by determining a region where a plurality of normalsto said surface converge.

Example 63: The display system of any of Examples 60, 61, or 62, whereinsaid processing electronics is configured to fit said surface to saidlocus of points to obtain said surface.

Example 64: The display system of any of the Examples above, whereinsaid processing electronics is configured to use a render camera torender virtual images to be presented to the eye of the user, saidrender camera having a position determined by said center of rotation.

Example 65: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content to the user's vision field at differentamounts of at least one of divergence and collimation and thus thedisplayed virtual image content appears to originate from differentdepths.

Example 66: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising:

-   -   a frame configured to be supported on a head of the user;    -   a head-mounted display disposed on the frame, said display        configured to project light into said user's eye to display        virtual image content to the user's vision field;    -   an eye tracking camera configured to image the user's eye;    -   a plurality of light emitters; and    -   processing electronics in communication with the display and the        eye tracking camera, the processing electronics configured to:        -   receive images of the user's eye captured by the eye            tracking camera, glint reflections of the different light            emitters observable in said images of the eye captured by            the eye tracking camera; and        -   estimate a location of a center of rotation of the user's            eye based on the location of the glint reflections in said            images produced by said eye tracking camera and based on the            location of the eye tracking camera and the locations of the            emitters that produced said glint reflections for multiple            eye poses.

Example 67: The system of Example 66, wherein to obtain an estimate ofthe center of rotation of said eye, the processing electronics areconfigured to:

-   -   determine a plurality of estimates of the center of corneal        curvature of the user's eye based a plurality of glint        reflections for multiple eye poses; and    -   determine the estimate of the center of rotation of the user's        eye based on the plurality of estimates of the center of corneal        curvature of the user's eye for said multiple eye poses.

Example 68: The system of Example 67, wherein to determine saidplurality of estimates of the corneal curvature of the user's eye, theprocessing electronics are configured to:

-   -   determine a first direction toward the center of corneal        curvature based on at least a respective location of a portion        of said plurality of emitters and a location of the eye tracking        camera;    -   determine a second direction toward the center of corneal        curvature based on at least a respective location of at least a        portion of said plurality of emitters and a location of the eye        tracking camera; and    -   determine an estimate of the center of corneal curvature of the        user's eye based on said the first and second directions.

Example 69: The display system of Example 68, wherein said processingelectronics is configured to determine the first direction by:

-   -   defining a first plane that includes the location of the eye        tracking camera, a location of a first glint reflection and a        location of the light emitter corresponding to said first glint        reflection;    -   defining a second plane that includes the location of the eye        tracking camera, a location of a second glint reflection and a        location of the light emitter corresponding to said second glint        reflection; and    -   determining a region of convergence of the first plane and the        second plane, the region of convergence extending along the        first direction.

Example 70: The display system of Example 69, said processingelectronics are configured to determine the second direction by:

-   -   defining a third plane that includes the location of the eye        tracking camera, the location of a third glint reflection, and a        location of the light emitter corresponding to said third glint        reflection;    -   defining a fourth plane that includes the location of the eye        tracking camera, the location of a fourth glint reflection, and        a location of the light emitter corresponding to said fourth        glint reflection; and    -   determining a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        second direction.

Example 71: The system of any of Examples 68-70, wherein to determinesaid plurality of estimates of the corneal curvature of the user's eye,the processing electronics are configured to:

-   -   determine a region of convergence between the first direction        and second direction to determine an estimate of the center of        corneal curvature of the user's eye.

Example 72: The system of any of Examples 19-71, wherein to obtain anestimate of the center of rotation of said eye, the processingelectronics are configured to:

-   -   generate a three-dimensional surface associated with the        plurality of estimates of the center of the corneal curvature;        and    -   determine the estimate of the center of rotation of the user's        eye based on the three-dimensional surface.

Example 73: The system of Example 72, wherein to generate athree-dimensional surface associated with the plurality of estimates ofthe center of the corneal curvature, the processing electronics areconfigured to fit a surface to the plurality of estimates of the centerof the corneal curvature.

Example 74: The system of Example 73, wherein to generate athree-dimensional surface associated with the plurality of estimates ofthe center of the corneal curvature, the processing electronics areconfigured to fit a sphere to the plurality of estimates of the centerof the corneal curvature.

Example 75: The system of any of Examples 72-74, wherein to determinethe estimate of the center of rotation of the user's eye, the processingelectronics are configured to:

-   -   determine two or more normals to the three-dimensional surface;        and    -   determine a region of convergence of the two or more normals,        wherein the region of convergence comprises the estimate of the        center of rotation of the user's eye.

Example 76: The system of any of Examples 69-75, wherein the one or moreimages of the user's eye comprise one or more images associated withdifferent gaze vectors of the user's eye.

Example 77: The system of any of Examples 69-76, wherein the processingelectronics are configured to map the cornea of the user's eye using agaze target.

Example 78: The display system of any of Examples 66-77, wherein saidprocessing electronics is configured to use a render camera to rendervirtual images to be presented to the eye of the user, said rendercamera having a position determined by said center of rotation.

Example 79: The display system of any of Examples 66-78, wherein saiddisplay is configured to project light into said user's eye to displayvirtual image content to the user's vision field at different amounts ofat least one of divergence and collimation and thus the displayedvirtual image content appears to originate from different depths.

Example 80: A method of determining one or more parameters associatedwith an eye for rendering virtual image content in a display systemconfigured to project light to an eye of a user to display the virtualimage content in a vision field of said user, said eye having a cornea,said cornea having a center of curvature, said method comprising:

-   -   with an eye tracking camera configured to image the eye of the        user and a plurality of light emitters disposed with respect to        said eye to form glints thereon, capturing a plurality of images        of the eye of the user, said images comprising a plurality of        glints; and    -   obtaining an estimate of a center of rotation of said eye based        on the plurality of glints, wherein obtaining an estimate of the        center of rotation of said eye comprises:        -   determining a plurality of estimates of the center of            corneal curvature of the user's eye based on the plurality            of glints;        -   generating a three-dimensional surface from the plurality of            estimates of the center of the corneal curvature; and        -   determining the estimate of the center of rotation of the            user's eye using the three-dimensional surface.

Example 81: The method of Example 80, wherein determining the pluralityof estimates of the corneal curvature of the user's eye comprises:

-   -   determining a first vector directed toward the center of corneal        curvature based on the locations of at least a portion of the        plurality of light emitters and a location of the eye tracking        camera;    -   determining a second vector directed toward the center of        corneal curvature based on locations of at least a portion of        the plurality of light emitters and a location of the eye        tracking camera; and    -   determining a region of convergence between the first vector and        second vector to determine an estimate of the center of corneal        curvature of the user's eye.

Example 82: The method of Example 81, wherein the first direction isdetermined by:

-   -   defining a first plane that includes the location of the eye        tracking camera, a location of a first glint reflection and a        location of the light emitter corresponding to said first glint        reflection,    -   defining a second plane that includes the location of the eye        tracking camera, a location of a second glint reflection and a        location of the light emitter corresponding to said second glint        reflection; and    -   determining a region of convergence of the first plane and the        second plane, the region of convergence extending along the        first direction.

Example 83: The method of Example 82, wherein the second direction isdetermined by:

-   -   defining a third plane that includes the location of the eye        tracking camera, the location of a third glint reflection, and a        location of the light emitter corresponding to said third glint        reflection;    -   defining a fourth plane that includes the location of the eye        tracking camera, the location of a fourth glint reflection, and        a location of the light emitter corresponding to said fourth        glint reflection; and    -   determining a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        second direction.

Example 84: The method of any of Examples 81-83, wherein generating athree-dimensional surface from the plurality of estimates of the centerof the corneal curvature comprises fitting a surface to the plurality ofestimates of the center of the corneal curvature.

Example 85: The method of any of Examples 81-83, wherein generating athree-dimensional surface from the plurality of estimates of the centerof the corneal curvature comprises fitting a sphere to the plurality ofestimates of the center of the corneal curvature.

Example 86: The method of any of Examples 81-85, wherein determining theestimate of the center of rotation of the user's eye comprises:

-   -   determining two or more vectors normal to the three-dimensional        surface; and    -   determining a region of convergence of the two or more vectors        normal to the three-dimensional surface, wherein the region of        convergence comprises the estimate of the center of rotation of        the user's eye.

Example 87: The method of any of Examples 81-86, wherein the pluralityof images of the user's eye comprise images associated with differentgaze directions of the user's eye.

Example 88: The method of any of Examples 81-87, further comprisingmapping the cornea of the user's eye using a gaze target.

Example 89: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising:

-   -   a frame configured to be supported on a head of the user;    -   a head-mounted display disposed on the frame, said display        configured to project light into said user's eye to display        virtual image content;    -   an eye tracking camera configured to image the user's eye; and    -   processing electronics in communication with the display and the        eye tracking camera, the processing electronics configured to:        -   receive multiple pairs of captured images of the user's eye            from the eye tracking camera;        -   for pairs of images received from the eye tracking camera,            respectively, obtain an estimate of a center of corneal            curvature of the user's eye based at least in part on the            respective pair of captured images;        -   determine a three-dimensional surface based on the estimated            centers of corneal curvature of the user's eye obtained            based on the multiple pairs of captured images of the user's            eye received from the eye tracking camera; and        -   identify a center of curvature of the 3D surface to obtain            an estimate of a center of rotation of the user's eye.

Example 90: The display system of Example 89, wherein said processingelectronics is configured to fit a three-dimensional surface to theestimated centers of corneal curvature of the user's eye obtained basedon the multiple pairs of captured images of the user's eye received fromthe eye tracking camera.

Example 91: The display system of Examples 89 or 90, wherein to obtainthe estimate of the center of corneal curvature of the user's eye basedat least in part on the respective pair of captured images, theprocessing electronics are configured to:

-   -   determine a first vector along which the center of corneal        curvature of the user's eye is estimated to be located based on        a first image received from a location of the eye tracking        camera;    -   determine a second vector along which the center of corneal        curvature of the user's eye is estimated to be located based on        a second image received from a location of the eye tracking        camera, the first and second images corresponding to one of said        pairs of images; and    -   identify a region of convergence between paths extending in the        direction of the first vector and the second vector to obtain an        estimate of a center of corneal curvature of the user's eye.

Example 92: The display system of Example 91, further comprising:

-   -   a plurality of light emitters configured to illuminate the        user's eye to form glint reflections thereon,    -   wherein to determine the first vector based on the first image        of the pair of captured images, the processing electronics are        configured to:        -   define a first plane that includes the location of the eye            tracking camera, a location of a first glint reflection and            a location of the light emitter corresponding to said first            glint reflection;        -   define a second plane that includes the location of the eye            tracking camera, a location of a second glint reflection and            a location of the light emitter corresponding to said second            glint reflection; and        -   identify a region of convergence of the first plane and the            second plane, the region of convergence extending along the            direction of the first vector.

Example 93: The display system of Example 92, wherein to determine thesecond vector based on the second image in each pair of captured images,the processing electronics are configured to:

-   -   define a third plane that includes the location of the eye        tracking camera, the location of a third glint reflection, and a        location of the light emitter corresponding to said third glint        reflection;    -   define a fourth plane that includes the location of the eye        tracking camera, the location of a fourth glint reflection, and        a location of the light emitter corresponding to said fourth        glint reflection; and    -   determine a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        direction of the second vector.

Example 94: The display system of any of Examples 89-93, wherein saidprocessing electronics is configured to use a render camera to rendervirtual images to be presented to the eye of the user, said rendercamera having a position determined by said center of rotation.

Example 95: The display system of any of Examples 89-94, wherein saiddisplay is configured to project light into said user's eye to displayvirtual image content to the user's vision field at different amounts ofat least one of divergence and collimation and thus the displayedvirtual image content appears to originate from different depths.

Example 96: The display system of any of the Examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 97: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising:

-   -   a frame configured to be supported on a head of the user;    -   a head-mounted display disposed on the frame, said display        configured to project light into said user's eye to display        virtual image content to the user's vision field;    -   at least one eye tracking camera configured to image the user's        eye;    -   a plurality of light emitters; and    -   processing electronics in communication with the display and the        eye tracking camera, the processing electronics configured to:        -   receive images of the user's eye captured by the at least            one eye tracking camera at a first and second location,            glint reflections of the different light emitters observable            in said images of the eye captured by the eye tracking            camera; and        -   estimate a location of said center of corneal curvature of            the user's eye based on the location of the glint            reflections in said images produced by said at least one eye            tracking camera and based on the location of the at least            one eye tracking camera and the locations of the emitters            that produced said respective glint reflections.

Example 98: The display system of Example 97, wherein said processingelectronics is configured to:

-   -   based on the location of the glint reflections in one or more        images produced by said at least one eye tracking camera and        based on the first location of the at least one eye tracking        camera and the location of the emitters that produced said glint        reflections, determine a first direction toward the center of        corneal curvature of the user's eye; and    -   based on the location of the glint reflections in one or more        images produced by said at least one eye tracking camera and        based on the second location of the at least one eye tracking        camera and the location of the emitters that produced said glint        reflections, determine a second direction toward the center of        corneal curvature of the user's eye.

Example 99: The display system of Example 98, wherein said processingelectronics is configured to determine the first direction by:

-   -   defining a first plane that includes the first location of the        at least one eye tracking camera, a location of a first glint        reflection and a location of the light emitter corresponding to        said first glint reflection;    -   defining a second plane that includes the first location of the        at least one eye tracking camera, a location of a second glint        reflection and a location of the light emitter corresponding to        said second glint reflection; and    -   determining a region of convergence of the first plane and the        second plane, the region of convergence extending along the        first direction.

Example 100: The display system of Example 99, said processingelectronics are configured to determine the second direction by:

-   -   defining a third plane that includes the second location of the        at least one eye tracking camera, the location of a third glint        reflection, and a location of the light emitter corresponding to        said third glint reflection;    -   defining a fourth plane that includes the second location of the        at least one eye tracking camera, the location of a fourth glint        reflection, and a location of the light emitter corresponding to        said fourth glint reflection; and    -   determining a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        second direction.

Example 101: The display system of any of the Examples above, whereinsaid processing electronics is configured to estimate a location of saidcenter of corneal curvature of the user's eye based on said first andsecond directions toward the center of the corneal curvature of theuser's eye.

Example 102: The display system of any of the Examples above, whereinsaid processing electronics is configured to:

-   -   determine said first direction along which the center of corneal        curvature of the user's eye is estimated to be located based on        at least one first image received from the first location of the        at least one eye tracking camera; and    -   determine said second direction along which the center of        corneal curvature of the user's eye is estimated to be located        based on at least one second image received from the second        location of the at least one eye tracking camera, said first and        second directions converging toward a region.

Example 103: The display system of any of the Examples above, whereinsaid processing electronics is configured to:

-   -   obtain an estimate of a center of corneal curvature of the        user's eye based on the convergence of the first and second        directions.

Example 104: The display system of any of the Examples above, whereinsaid processing electronics is configured to estimate a location of saidcenter of corneal curvature of the user's eye by identifying a region ofconvergence of said first and second directions toward the center of thecorneal curvature of the user's eye.

Example 105: The display system of any of the Examples above, whereinsaid processing electronics is configured to obtain an estimate of acenter of rotation of the user's eye based on multiple determinations ofthe center of corneal curvature of the user's eye for different eyeposes.

Example 106: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a locus of pointscorresponding to estimates of the center of corneal curvature of theuser's eye for different eye poses.

Example 107: The display system of Example 106, wherein said processingelectronics is configured to obtain an estimate of a center of rotationof the user's eye based on said locus of points corresponding toestimates of the center of corneal curvature of the user's eye fordifferent eye poses.

Example 108: The display system of Examples 106 or 107, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye.

Example 109: The display system of Examples 106 or 107, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye by estimating a center of curvature of said surface.

Example 110: The display system of Examples 106 or 107, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye by determining a region where a plurality of normalsto said surface converge.

Example 111: The display system of any of Examples 108, 109, or 110,wherein said processing electronics is configured to fit said surface tosaid locus of points to obtain said surface.

Example 112: The display system of any of the Examples above, whereinsaid processing electronics is configured to use a render camera torender virtual images to be presented to the eye of the user, saidrender camera having a position determined by said center of rotation.

Example 113: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content to the user's vision field at differentamounts of at least one of divergence and collimation and thus thedisplayed virtual image content appears to originate from differentdepths.

Example 115: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content to the user's vision field such that thedisplayed virtual image content appears to originate from differentdepths.

Example 116: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content to the user's vision field at differentamounts of divergence such that the displayed virtual image contentappears to originate from different depths.

Example 117: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye thatdivergences and to project light into said user's eye that is collimatedto display virtual image content to the user's vision field that appearsto originate from different depths.

Part III

Example 1: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising:

a frame configured to be supported on a head of the user;

a head-mounted display disposed on the frame, said display configured toproject light into said user's eye to display virtual image content tothe user's vision field;

first and second eye tracking cameras configured to image the user'seye;

a plurality of light emitters; and

processing electronics in communication with the display and the firstand second eye tracking cameras, the processing electronics configuredto:

-   -   receive images of the user's eye captured by the first and        second eye tracking cameras, glint reflections of the different        light emitters observable in said images of the eye captured by        the first and second eye tracking cameras; and    -   estimate a parameter of the eye based on the location of the        glint reflections in said images produced by both said first and        second eye tracking camera and based on the location of both the        first and second eye tracking cameras and the locations of the        emitters that produced said respective glint reflections.

Example 2: The display system of Example 1, wherein said processingelectronics is configured to estimate said parameter of the eye by:

-   -   based on the location of the glint reflections in one or more        images produced by said first eye tracking camera and based on        the location of the first eye tracking camera and the location        of the emitters that produced said glint reflections, determine        a first direction; and    -   based on the location of the glint reflections in one or more        images produced by said second eye tracking camera and based on        the location of the second eye tracking camera and the location        of the emitters that produced said glint reflections, determine        a second direction.

Example 3: The display system of Example 2, wherein said processingelectronics is configured to determine the first direction by:

-   -   defining a first plane that includes the first eye tracking        camera, a location of a first glint reflection and a location of        the light emitter corresponding to said first glint reflection;    -   defining a second plane that includes the first eye tracking        camera, a location of a second glint reflection and a location        of the light emitter corresponding to said second glint        reflection; and    -   determining a region of convergence of the first plane and the        second plane, the region of convergence extending along the        first direction.

Example 4: The display system of Example 3, said processing electronicsare configured to determine the second direction by:

-   -   defining a third plane that includes the second eye tracking        camera, the location of a third glint reflection, and a location        of the light emitter corresponding to said third glint        reflection;    -   defining a fourth plane that includes the second eye tracking        camera, the location of a fourth glint reflection, and a        location of the light emitter corresponding to said fourth glint        reflection; and    -   determining a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        second direction.

Example 5: The display system of any of the Examples above, wherein saidprocessing electronics is configured to estimate a location of saidparameter of the user's eye based on said first and second directions.

Example 6: The display system of any of the Examples above, wherein saidprocessing electronics is configured to:

-   -   determine said first direction based on at least one first image        received from the first eye tracking camera; and    -   determine said second direction based on at least one second        image received from the second eye tracking camera, said first        and second directions converging toward a region.

Example 7: The display system of any of the Examples above, wherein saidprocessing electronics is configured to:

-   -   obtain an estimate said parameter based on the convergence of        the first and second directions.

Example 8: The display system of any of the Examples above, wherein saidprocessing electronics is configured to estimate said parameter byidentifying a region of convergence of said first and second directions.

Example 9: The display system of any of the Examples above, wherein saidprocessing electronics is configured to obtain an estimate of anadditional parameter of the user's eye based on multiple determinationsof the said other parameter of the user's eye for different eye poses.

Example 10: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a locus of pointscorresponding to estimates of the parameter of the user's eye fordifferent eye poses.

Example 11: The display system of Example 10, wherein said processingelectronics is configured to obtain an estimate of an additionalparameter of the user's eye based on said locus of points correspondingto estimates of the other parameter of the user's eye for different eyeposes.

Example 12: The display system of Examples 10 or 11, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of said additionalparameter of the user's eye.

Example 13: The display system of Examples 10 or 11, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of said additionalparameter of the user's eye by estimating a center of curvature of saidsurface.

Example 14: The display system of Examples 10 or 11, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of said additionalparameter of the user's eye by determining a region where a plurality ofnormals to said surface converge.

Example 15: The display system of any of Examples 12, 13, or 14, whereinsaid processing electronics is configured to fit said surface to saidlocus of points to obtain said surface.

Example 16: The display system of any of the Examples above, whereinsaid processing electronics is configured to use a render camera torender virtual images to be presented to the eye of the user, saidrender camera having a position determined by said additional parameter.

Example 17: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content to the user's vision field at differentamounts of at least one of divergence and collimation and thus thedisplayed virtual image content appears to originate from differentdepths.

Example 18: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising:

-   -   a frame configured to be supported on a head of the user;    -   a head-mounted display disposed on the frame, said display        configured to project light into said user's eye to display        virtual image content to the user's vision field;    -   first and second eye tracking cameras configured to image the        user's eye;    -   a plurality of light emitters; and    -   processing electronics in communication with the display and the        first and second eye tracking cameras, the processing        electronics configured to:        -   receive images of the user's eye captured by the first and            second eye tracking cameras, glint reflections of the            different light emitters observable in said images of the            eye captured by the first and second eye tracking cameras;            and        -   estimate a location of a first parameter of the user's eye            based on the location of the glint reflections in said            images produced by both said first and second eye tracking            camera and based on said the location of both the first and            second eye tracking cameras and the locations of the            emitters that produced said glint reflections for multiple            eye poses.

Example 19: The display system of Example 18, wherein to obtain anestimate of the first parameter of said eye, the processing electronicsare configured to:

-   -   determine a plurality of estimates of a second parameter of the        user's eye based a plurality of glint reflections for multiple        eye poses; and    -   determine the estimate of the first parameter of the user's eye        based on the plurality of estimates of the second parameter of        the user's eye for said multiple eye poses.

Example 20: The display system of Example 19, wherein to determine saidplurality of estimates of the second parameter of the user's eye, theprocessing electronics are configured to:

-   -   determine a first direction based on the respective locations of        at least a portion of said plurality of emitters and a first        camera of the eye tracking cameras;    -   determine a second direction based on the respective locations        of at least a portion of said plurality of emitters and a second        camera of the eye tracking cameras; and    -   determine an estimate of the second parameter of the user's eye        based on said the first and second directions.

Example 21: The display system of Example 20, wherein said processingelectronics is configured to determine the first direction by:

-   -   defining a first plane that includes the first eye tracking        camera, a location of a first glint reflection and a location of        the light emitter corresponding to said first glint reflection;    -   defining a second plane that includes the first eye tracking        camera, a location of a second glint reflection and a location        of the light emitter corresponding to said second glint        reflection; and    -   determining a region of convergence of the first plane and the        second plane, the region of convergence extending along the        first direction.

Example 22: The display system of Example 21, said processingelectronics are configured to determine the second direction by:

-   -   defining a third plane that includes the second eye tracking        camera, the location of a third glint reflection, and a location        of the light emitter corresponding to said third glint        reflection;    -   defining a fourth plane that includes the second eye tracking        camera, the location of a fourth glint reflection, and a        location of the light emitter corresponding to said fourth glint        reflection; and    -   determining a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        second direction.

Example 23: The display system of any of Examples 20-22, wherein todetermine said plurality of estimates of second parameter of the user'seye, the processing electronics are configured to:

-   -   determine a region of convergence between the first direction        and second direction to determine an estimate of the second        parameter of the user's eye.

Example 24: The display system of any of Examples 19-23, wherein toobtain an estimate of said first parameter of said eye, the processingelectronics are configured to:

-   -   generate a three-dimensional surface associated with the        plurality of estimates of the second parameter of the eye; and    -   determine the estimate of the first parameter of the user's eye        based on the three-dimensional surface.

Example 25: The display system of Example 24, wherein to generate athree-dimensional surface associated with the plurality of estimates ofthe second parameter of the eye, the processing electronics areconfigured to fit a surface to the plurality of estimates of the secondparameter.

Example 26: The display system of Example 24, wherein to generate athree-dimensional surface associated with the plurality of estimates ofthe second parameter, the processing electronics are configured to fit aspherical surface to the plurality of estimates of the second parameter.

Example 27: The display system of any of Example 24-26, wherein todetermine the estimate of said first parameter of the user's eye, theprocessing electronics are configured to:

-   -   determine two or more normals to the three-dimensional surface;        and    -   determine a region of convergence of the two or more normals,        wherein the region of convergence comprises the estimate of the        first parameter of the user's eye.

Example 28: The display system of any of Examples 21-27, wherein the oneor more images of the user's eye comprise one or more images associatedwith different gaze vectors of the user's eye.

Example 29: The display system of any of Examples 21-28, wherein theprocessing electronics are configured to use a gaze target.

Example 30: The display system of any of Examples 18-29, wherein saidprocessing electronics is configured to use a render camera to rendervirtual images to be presented to the eye of the user, said rendercamera having a position determined by said first parameter of said eye.

Example 31: The display system of any of Examples 18-30, wherein saiddisplay is configured to project light into said user's eye to displayvirtual image content to the user's vision field at different amounts ofat least one of divergence and collimation and thus the displayedvirtual image content appears to originate from different depths.

Example 32: A method of determining one or more parameters associatedwith an eye for rendering virtual image content in a display systemconfigured to project light to an eye of a user to display the virtualimage content in a vision field of said user, said eye having a cornea,said cornea having a center of curvature, said method comprising:

-   -   with a plurality of eye tracking cameras configured to image the        eye of the user and a plurality of light emitters disposed with        respect to said eye to form glints thereon, capturing a        plurality of images of the eye of the user, said images        comprising a plurality of glints; and    -   obtaining a first parameter of said eye based on the plurality        of glints, wherein obtaining a first parameter of said eye        comprises:        -   determining a plurality of estimates of a second parameter            of the user's eye based on the plurality of glints;        -   generating a three-dimensional surface from the plurality of            estimates of the second parameter; and        -   determining the estimate of the first parameter of the            user's eye using the three-dimensional surface.

Example 33: The method of Example 32, wherein determining the pluralityof estimates of the second parameter of the user's eye comprises:

-   -   determining a first vector directed based on the locations of at        least a portion of the plurality of light emitters and the        location of a first camera of the plurality of eye tracking        cameras;    -   determining a second vector directed based on locations of at        least a portion of the plurality of light emitters and the        location of a second camera of the plurality of eye tracking        cameras; and    -   determining a region of convergence between the first vector and        second vector to determine an estimate of the second parameter        of the user's eye.

Example 34: The method of Example 33, wherein the first direction isdetermined by:

-   -   defining a first plane that includes the first eye tracking        camera, a location of a first glint reflection and a location of        the light emitter corresponding to said first glint reflection,    -   defining a second plane that includes the first eye tracking        camera, a location of a second glint reflection and a location        of the light emitter corresponding to said second glint        reflection; and    -   determining a region of convergence of the first plane and the        second plane, the region of convergence extending along the        first direction.

Example 35: The method of Example 33, wherein the second direction isdetermined by:

-   -   defining a third plane that includes the second eye tracking        camera, the location of a third glint reflection, and a location        of the light emitter corresponding to said third glint        reflection;    -   defining a fourth plane that includes the second eye tracking        camera, the location of a fourth glint reflection, and a        location of the light emitter corresponding to said fourth glint        reflection; and    -   determining a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        second direction.

Example 36: The method of any of Examples 32-35, wherein generating athree-dimensional surface from the plurality of estimates of the secondparameter comprises fitting a surface to the plurality of estimates ofthe second parameter.

Example 37: The method of any of Examples 32-35, wherein generating athree-dimensional surface from the plurality of estimates of the secondparameter comprises fitting a sphere to the plurality of estimates ofthe second parameter.

Example 38: The method of any of Examples 32-37, wherein determining theestimate of the first parameter of the user's eye comprises:

-   -   determining two or more vectors normal to the three-dimensional        surface; and    -   determining a region of convergence of the two or more vectors        normal to the three-dimensional surface, wherein the region of        convergence comprises the estimate of the first parameter of the        user's eye.

Example 39: The method of any of Examples 32-38, wherein the pluralityof images of the user's eye comprise images associated with differentgaze directions of the user's eye.

Example 40: The method of any of Examples 32-39, further comprisingusing a gaze target.

Example 41: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising:

-   -   a frame configured to be supported on a head of the user;    -   a head-mounted display disposed on the frame, said display        configured to project light into said user's eye to display        virtual image content;    -   first and second eye tracking cameras configured to image the        user's eye; and    -   processing electronics in communication with the display and the        first and second eye tracking cameras, the processing        electronics configured to:        -   receive multiple pairs of captured images of the user's eye            from the first and second eye tracking cameras;        -   for pairs of images received from the first and second eye            tracking cameras, respectively, obtain an estimate of a            second parameter of the user's eye based at least in part on            the respective pair of captured images;        -   determine a three-dimensional surface based on the estimated            second parameters of the user's eye obtained based on the            multiple pairs of captured images of the user's eye received            from the respective first and second eye tracking cameras;            and        -   identify a center of curvature of the 3D surface to obtain            an estimate of a first parameter of the user's eye.

Example 42: The display system of Example 41, wherein said processingelectronics is configured to fit a three-dimensional surface to theestimated second parameters of the user's eye obtained based on themultiple pairs of captured images of the user's eye received from therespective first and second eye tracking cameras.

Example 43: The display system of Examples 41 or 42, wherein to obtainthe estimate of the second parameter of the user's eye based at least inpart on the respective pair of captured images, the processingelectronics are configured to:

-   -   determine a first vector based on a first image received from        the first eye tracking camera;    -   determine a second vector based on a second image received from        the second eye tracking camera, the first and second images        corresponding to one of said pairs of images; and    -   identify a region of convergence between paths extending in the        direction of the first vector and the second vector to obtain an        estimate of the second parameter of the user's eye.

Example 44: The display system of Example 43, further comprising:

-   -   a plurality of light emitters configured to illuminate the        user's eye to form glint reflections thereon,    -   wherein to determine the first vector based on the first image        of the pair of captured images, the processing electronics are        configured to:        -   define a first plane that includes the first eye tracking            camera, a location of a first glint reflection and a            location of the light emitter corresponding to said first            glint reflection;        -   define a second plane that includes the first eye tracking            camera, a location of a second glint reflection and a            location of the light emitter corresponding to said second            glint reflection; and        -   identify a region of convergence of the first plane and the            second plane, the region of convergence extending along the            direction of the first vector.

Example 45: The display system of Example 44, wherein to determine thesecond vector based on the second image in each pair of captured images,the processing electronics are configured to:

-   -   define a third plane that includes the second eye tracking        camera, the location of a third glint reflection, and a location        of the light emitter corresponding to said third glint        reflection;    -   define a fourth plane that includes the second eye tracking        camera, the location of a fourth glint reflection, and a        location of the light emitter corresponding to said fourth glint        reflection; and    -   determine a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        direction of the second vector.

Example 46: The display system of any of Examples 41-45, wherein saidprocessing electronics is configured to use a render camera to rendervirtual images to be presented to the eye of the user, said rendercamera having a position determined by said first parameter.

Example 47: The display system of any of Examples 41-46, wherein saiddisplay is configured to project light into said user's eye to displayvirtual image content to the user's vision field at different amounts ofdivergence such that the displayed virtual image content appears tooriginate from different depths or wherein said display is configured toproject light into said user's eye that divergences and to project lightinto said user's eye that is collimated to display virtual image contentto the user's vision field that appears to originate from differentdepths.

Example 48: The display system of any of the Examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 49: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising:

-   -   a frame configured to be supported on a head of the user;    -   a head-mounted display disposed on the frame, said display        configured to project light into said user's eye to display        virtual image content to the user's vision field;    -   an eye tracking camera configured to image the user's eye;    -   a plurality of light emitters; and    -   processing electronics in communication with the display and the        eye tracking camera, the processing electronics configured to:        -   receive images of the user's eye captured by the eye            tracking camera, glint reflections of the different light            emitters observable in said images of the eye captured by            the eye tracking camera; and        -   estimate a location of a parameter of the user's eye based            on the location of the glint reflections in said images            produced by said eye tracking camera and based on the            location of the eye tracking camera and the locations of the            emitters that produced said respective glint reflections.

Example 50: The display system of Example 49, wherein said processingelectronics is configured to:

-   -   based on the location of the glint reflections in one or more        images produced by said eye tracking camera and based on the        location of the eye tracking camera and the location of the        emitters that produced said glint reflections, determine a first        direction; and    -   based on the location of the glint reflections in one or more        images produced by said eye tracking camera and based on the        location of the eye tracking camera and the location of the        emitters that produced said glint reflections, determine a        second direction.

Example 51: The display system of Example 50, wherein said processingelectronics is configured to determine the first direction by:

-   -   defining a first plane that includes the location of the eye        tracking camera, a location of a first glint reflection and a        location of the light emitter corresponding to said first glint        reflection;    -   defining a second plane that includes the location of the eye        tracking camera, a location of a second glint reflection and a        location of the light emitter corresponding to said second glint        reflection; and    -   determining a region of convergence of the first plane and the        second plane, the region of convergence extending along the        first direction.

Example 52: The display system of Example 51, said processingelectronics are configured to determine the second direction by:

-   -   defining a third plane that includes the location of the eye        tracking camera, the location of a third glint reflection, and a        location of the light emitter corresponding to said third glint        reflection;    -   defining a fourth plane that includes the location of the eye        tracking camera, the location of a fourth glint reflection, and        a location of the light emitter corresponding to said fourth        glint reflection; and    -   determining a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        second direction.

Example 53: The display system of any of the Examples above, whereinsaid processing electronics is configured to estimate a location of saidparameter of the user's eye based on said first and second directions.

Example 54: The display system of any of the Examples above, whereinsaid processing electronics is configured to:

-   -   determine said first direction based on at least one first image        received from the location of the eye tracking camera; and    -   determine said second direction based on at least one second        image received from the location of the eye tracking camera,        said first and second directions converging toward a region.

Example 55: The display system of any of the Examples above, whereinsaid processing electronics is configured to:

-   -   obtain an estimate of said parameter of the user's eye based on        the convergence of the first and second directions.

Example 56: The display system of any of the Examples above, whereinsaid processing electronics is configured to estimate said parameter ofthe user's eye by identifying a region of convergence of said first andsecond directions.

Example 57: The display system of any of the Examples above, whereinsaid processing electronics is configured to obtain an estimate of anaddition parameter based on multiple determinations of the parameter theuser's eye for different eye poses.

Example 58: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a locus of pointscorresponding to estimates of the parameter of the user's eye fordifferent eye poses.

Example 59: The display system of Example 58, wherein said processingelectronics is configured to obtain an estimate of an additionalparameter of the user's eye based on said locus of points correspondingto estimates of the other parameter of the user's eye for different eyeposes.

Example 60: The display system of Examples 58 or 59, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of the additionalparameter of the user's eye.

Example 61: The display system of Examples 58 or 59, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a additional parameterof the user's eye by estimating a center of curvature of said surface.

Example 62: The display system of Examples 58 or 59, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of an additionalparameter of the user's eye by determining a region where a plurality ofnormals to said surface converge.

Example 63: The display system of any of Examples 60, 61, or 62, whereinsaid processing electronics is configured to fit said surface to saidlocus of points to obtain said surface.

Example 64: The display system of any of the Examples above, whereinsaid processing electronics is configured to use a render camera torender virtual images to be presented to the eye of the user, saidrender camera having a position determined by said additional parameter.

Example 65: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content to the user's vision field at differentamounts of divergence such that the displayed virtual image contentappears to originate from different depths or wherein said display isconfigured to project light into said user's eye that divergences and toproject light into said user's eye that is collimated to display virtualimage content to the user's vision field that appears to originate fromdifferent depths.

Example 66: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising:

-   -   a frame configured to be supported on a head of the user;    -   a head-mounted display disposed on the frame, said display        configured to project light into said user's eye to display        virtual image content to the user's vision field;    -   an eye tracking camera configured to image the user's eye;    -   a plurality of light emitters; and    -   processing electronics in communication with the display and the        eye tracking camera, the processing electronics configured to:        -   receive images of the user's eye captured by the eye            tracking camera, glint reflections of the different light            emitters observable in said images of the eye captured by            the eye tracking camera; and        -   estimate a location of a first parameter of the user's eye            based on the location of the glint reflections in said            images produced by said eye tracking camera and based on            said location of the eye tracking camera and the locations            of the emitters that produced said glint reflections for            multiple eye poses.

Example 67: The system of Example 66, wherein to obtain an estimate ofthe first parameter of said eye, the processing electronics areconfigured to:

-   -   determine a plurality of estimates of a second parameter of the        user's eye based a plurality of glint reflections for multiple        eye poses; and    -   determine the estimate of the first parameter of the user's eye        based on the plurality of estimates of the second parameter of        the user's eye for said multiple eye poses.

Example 68: The system of Example 67, wherein to determine saidplurality of estimates of the second parameter of the user's eye, theprocessing electronics are configured to:

-   -   determine a first direction based on at least a respective        location of a portion of said plurality of emitters and the        location of the eye tracking camera;    -   determine a second direction toward the center of corneal        curvature based on at least a respective location of at least a        portion of said plurality of emitters and the location of the        eye tracking camera; and    -   determine an estimate of the center of corneal curvature of the        user's eye based on said the first and second directions.

Example 69: The display system of Example 68, wherein said processingelectronics is configured to determine the first direction by:

-   -   defining a first plane that includes the location of the eye        tracking camera, a location of a first glint reflection and a        location of the light emitter corresponding to said first glint        reflection;    -   defining a second plane that includes the location of the eye        tracking camera, a location of a second glint reflection and a        location of the light emitter corresponding to said second glint        reflection; and    -   determining a region of convergence of the first plane and the        second plane, the region of convergence extending along the        first direction.

Example 70: The display system of Example 69, said processingelectronics are configured to determine the second direction by:

-   -   defining a third plane that includes the location of the eye        tracking camera, the location of a third glint reflection, and a        location of the light emitter corresponding to said third glint        reflection;    -   defining a fourth plane that includes the location of the eye        tracking camera, the location of a fourth glint reflection, and        a location of the light emitter corresponding to said fourth        glint reflection; and    -   determining a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        second direction.

Example 71: The system of any of Examples 68-70, wherein to determinesaid plurality of estimates of the second parameter of the user's eye,the processing electronics are configured to:

-   -   determine a region of convergence between the first direction        and second direction to determine an estimate of the second        parameter of the user's eye.

Example 72: The system of any of Examples 19-71, wherein to obtain anestimate of the first parameter of said eye, the processing electronicsare configured to:

-   -   generate a three-dimensional surface associated with the        plurality of estimates of the second parameter; and    -   determine the estimate of the first parameter of the user's eye        based on the three-dimensional surface.

Example 73: The system of Example 72, wherein to generate athree-dimensional surface associated with the plurality of estimates ofthe second parameter, the processing electronics are configured to fit asurface to the plurality of estimates of the first parameter.

Example 74: The system of Example 73, wherein to generate athree-dimensional surface associated with the plurality of estimates ofthe second parameter, the processing electronics are configured to fit asphere to the plurality of estimates of the second parameter.

Example 75: The system of any of Examples 72-74, wherein to determinethe estimate of the first parameter of the user's eye, the processingelectronics are configured to:

-   -   determine two or more normals to the three-dimensional surface;        and    -   determine a region of convergence of the two or more normals,        wherein the region of convergence comprises the estimate of the        first parameter of the user's eye.

Example 76: The system of any of Examples 69-75, wherein the one or moreimages of the user's eye comprise one or more images associated withdifferent gaze vectors of the user's eye.

Example 77: The system of any of Examples 69-76, wherein the processingelectronics are configured to use a gaze target.

Example 78: The display system of any of Examples 66-77, wherein saidprocessing electronics is configured to use a render camera to rendervirtual images to be presented to the eye of the user, said rendercamera having a position determined by said first parameter.

Example 79: The display system of any of Examples 66-78, wherein saiddisplay is configured to project light into said user's eye to displayvirtual image content to the user's vision field at different amounts ofdivergence such that the displayed virtual image content appears tooriginate from different depths or wherein said display is configured toproject light into said user's eye that divergences and to project lightinto said user's eye that is collimated to display virtual image contentto the user's vision field that appears to originate from differentdepths.

Example 80: A method of determining one or more parameters associatedwith an eye for rendering virtual image content in a display systemconfigured to project light to an eye of a user to display the virtualimage content in a vision field of said user, said eye having a cornea,said cornea having a center of curvature, said method comprising:

-   -   with an eye tracking camera configured to image the eye of the        user and a plurality of light emitters disposed with respect to        said eye to form glints thereon, capturing a plurality of images        of the eye of the user, said images comprising a plurality of        glints; and    -   obtaining an estimate of a first parameter of said eye based on        the plurality of glints, wherein obtaining an estimate of the        first parameter of said eye comprises:        -   determining a plurality of estimates of as second parameter            of the user's eye based on the plurality of glints;        -   generating a three-dimensional surface from the plurality of            estimates of the second parameter; and        -   determining the estimate of the first parameter of the            user's eye using the three-dimensional surface.

Example 81: The method of Example 80, wherein determining the pluralityof estimates of the second parameter of the user's eye comprises:

-   -   determining a first vector based on the locations of at least a        portion of the plurality of light emitters and the location of        the eye tracking camera;    -   determining a second vector based on locations of at least a        portion of the plurality of light emitters and the location of        the eye tracking camera; and    -   determining a region of convergence between the first vector and        second vector to determine an estimate of the second parameter        of the user's eye.

Example 82: The method of Example 81, wherein the first direction isdetermined by:

-   -   defining a first plane that includes the location of the eye        tracking camera, a location of a first glint reflection and a        location of the light emitter corresponding to said first glint        reflection,    -   defining a second plane that includes the location of the eye        tracking camera, a location of a second glint reflection and a        location of the light emitter corresponding to said second glint        reflection; and    -   determining a region of convergence of the first plane and the        second plane, the region of convergence extending along the        first direction.

Example 83: The method of Example 82, wherein the second direction isdetermined by:

-   -   defining a third plane that includes the location of the eye        tracking camera, the location of a third glint reflection, and a        location of the light emitter corresponding to said third glint        reflection;    -   defining a fourth plane that includes the location of the eye        tracking camera, the location of a fourth glint reflection, and        a location of the light emitter corresponding to said fourth        glint reflection; and    -   determining a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        second direction.

Example 84: The method of any of Examples 81-83, wherein generating athree-dimensional surface from the plurality of estimates of the secondparameter comprises fitting a surface to the plurality of estimates ofthe first parameter.

Example 85: The method of any of Examples 81-83, wherein generating athree-dimensional surface from the plurality of estimates of the secondparameter comprises fitting a sphere to the plurality of estimates ofthe second parameter.

Example 86: The method of any of Examples 81-85, wherein determining theestimate of the first parameter of the user's eye comprises:

-   -   determining two or more vectors normal to the three-dimensional        surface; and    -   determining a region of convergence of the two or more vectors        normal to the three-dimensional surface, wherein the region of        convergence comprises the estimate of the first parameter of the        user's eye.

Example 87: The method of any of Examples 81-86, wherein the pluralityof images of the user's eye comprise images associated with differentgaze directions of the user's eye.

Example 88: The method of any of Examples 81-87, further using a gazetarget.

Example 89: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising:

-   -   a frame configured to be supported on a head of the user;    -   a head-mounted display disposed on the frame, said display        configured to project light into said user's eye to display        virtual image content;    -   an eye tracking camera configured to image the user's eye; and    -   processing electronics in communication with the display and the        eye tracking camera, the processing electronics configured to:        -   receive multiple pairs of captured images of the user's eye            from the eye tracking camera;        -   for pairs of images received from the eye tracking camera,            respectively, obtain an estimate of a parameter of the            user's eye based at least in part on the respective pair of            captured images;        -   determine a three-dimensional surface based on the estimated            parameter of the user's eye obtained based on the multiple            pairs of captured images of the user's eye received from the            eye tracking camera; and        -   identify a center of curvature of the 3D surface to obtain            an estimate of an additional parameter of the user's eye.

Example 90: The display system of Example 89, wherein said processingelectronics is configured to fit a three-dimensional surface to theestimated parameter of the user's eye obtained based on the multiplepairs of captured images of the user's eye received from the eyetracking camera.

Example 91: The display system of Examples 89 or 90, wherein to obtainthe estimate of the parameter of the user's eye based at least in parton the respective pair of captured images, the processing electronicsare configured to:

-   -   determine a first vector based on a first image received from        the eye tracking camera;    -   determine a second vector based on a second image received from        the eye tracking camera, the first and second images        corresponding to one of said pairs of images; and    -   identify a region of convergence between paths extending in the        direction of the first vector and the second vector to obtain an        estimate of the parameter of the user's eye.

Example 92: The display system of Example 91, further comprising:

-   -   a plurality of light emitters configured to illuminate the        user's eye to form glint reflections thereon,    -   wherein to determine the first vector based on the first image        of the pair of captured images, the processing electronics are        configured to:        -   define a first plane that includes the location of the eye            tracking camera, a location of a first glint reflection and            a location of the light emitter corresponding to said first            glint reflection;        -   define a second plane that includes the location of the eye            tracking camera, a location of a second glint reflection and            a location of the light emitter corresponding to said second            glint reflection; and        -   identify a region of convergence of the first plane and the            second plane, the region of convergence extending along the            direction of the first vector.

Example 93: The display system of Example 92, wherein to determine thesecond vector based on the second image in each pair of captured images,the processing electronics are configured to:

-   -   define a third plane that includes the location of the eye        tracking camera, the location of a third glint reflection, and a        location of the light emitter corresponding to said third glint        reflection;    -   define a fourth plane that includes the location of the eye        tracking camera, the location of a fourth glint reflection, and        a location of the light emitter corresponding to said fourth        glint reflection; and    -   determine a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        direction of the second vector.

Example 94: The display system of any of Examples 89-93, wherein saidprocessing electronics is configured to use a render camera to rendervirtual images to be presented to the eye of the user, said rendercamera having a position determined by said additional parameter.

Example 95: The display system of any of Examples 89-94, wherein saiddisplay is configured to project light into said user's eye to displayvirtual image content to the user's vision field at different amounts ofdivergence such that the displayed virtual image content appears tooriginate from different depths or wherein said display is configured toproject light into said user's eye that divergences and to project lightinto said user's eye that is collimated to display virtual image contentto the user's vision field that appears to originate from differentdepths.

Example 96: The display system of any of the Examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 97: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising:

-   -   a frame configured to be supported on a head of the user;    -   a head-mounted display disposed on the frame, said display        configured to project light into said user's eye to display        virtual image content to the user's vision field;    -   at least one eye tracking camera configured to image the user's        eye;    -   a plurality of light emitters; and    -   processing electronics in communication with the display and the        eye tracking camera, the processing electronics configured to:        -   receive images of the user's eye captured by the at least            one eye tracking camera at a first and second location,            glint reflections of the different light emitters observable            in said images of the eye captured by the eye tracking            camera; and        -   estimate a location of a parameter of the user's eye based            on the location of the glint reflections in said images            produced by said at least one eye tracking camera and based            on the location of the at least one eye tracking camera and            the locations of the emitters that produced said respective            glint reflections.

Example 98: The display system of Example 97, wherein said processingelectronics is configured to:

-   -   based on the location of the glint reflections in one or more        images produced by said at least one eye tracking camera and        based on the first location of the at least one eye tracking        camera and the location of the emitters that produced said glint        reflections, determine a first direction; and    -   based on the location of the glint reflections in one or more        images produced by said at least one eye tracking camera and        based on the second location of the at least one eye tracking        camera and the location of the emitters that produced said glint        reflections, determine a second direction.

Example 99: The display system of Example 98, wherein said processingelectronics is configured to determine the first direction by:

-   -   defining a first plane that includes the first location of the        at least one eye tracking camera, a location of a first glint        reflection and a location of the light emitter corresponding to        said first glint reflection;    -   defining a second plane that includes the first location of the        at least one eye tracking camera, a location of a second glint        reflection and a location of the light emitter corresponding to        said second glint reflection; and    -   determining a region of convergence of the first plane and the        second plane, the region of convergence extending along the        first direction.

Example 100: The display system of Example 99, said processingelectronics are configured to determine the second direction by:

-   -   defining a third plane that includes the second location of the        at least one eye tracking camera, the location of a third glint        reflection, and a location of the light emitter corresponding to        said third glint reflection;    -   defining a fourth plane that includes the second location of the        at least one eye tracking camera, the location of a fourth glint        reflection, and a location of the light emitter corresponding to        said fourth glint reflection; and    -   determining a region of convergence of the third plane and the        fourth plane, the region of convergence extending along the        second direction.

Example 101: The display system of any of the Examples above, whereinsaid processing electronics is configured to estimate a location of saidparameter of the user's eye based on said first and second directions.

Example 102: The display system of any of the Examples above, whereinsaid processing electronics is configured to:

-   -   determine said first direction based on at least one first image        received from the first location of the at least one eye        tracking camera; and    -   determine said second direction based on at least one second        image received from the second location of the at least one eye        tracking camera, said first and second directions converging        toward a region.

Example 103: The display system of any of the Examples above, whereinsaid processing electronics is configured to:

-   -   obtain an estimate of said parameter based on the convergence of        the first and second directions.

Example 104: The display system of any of the Examples above, whereinsaid processing electronics is configured to estimate said parameter ofthe user's eye by identifying a region of convergence of said first andsecond directions.

Example 105: The display system of any of the Examples above, whereinsaid processing electronics is configured to obtain an estimate of anadditional parameter of the user's eye based on multiple determinationsof the other parameter of the user's eye for different eye poses.

Example 106: The display system of any of the Examples above, whereinsaid processing electronics is configured to determine a locus of pointscorresponding to estimates of the parameter of the user's eye fordifferent eye poses.

Example 107: The display system of Example 106, wherein said processingelectronics is configured to obtain an estimate of an additionalparameter of the user's eye based on said locus of points correspondingto estimates of the other parameter of the user's eye for different eyeposes.

Example 108: The display system of Examples 106 or 107, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of said parameter of theuser's eye.

Example 109: The display system of Examples 106 or 107, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of an additionalparameter of the user's eye by estimating a center of curvature of saidsurface.

Example 110: The display system of Examples 106 or 107, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate said parameter of theuser's eye by determining a region where a plurality of normals to saidsurface converge.

Example 111: The display system of any of Examples 108, 109, or 110,wherein said processing electronics is configured to fit said surface tosaid locus of points to obtain said surface.

Example 112: The display system of any of the Examples above, whereinsaid processing electronics is configured to use a render camera torender virtual images to be presented to the eye of the user, saidrender camera having a position determined by said additional parameter.

Example 113: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content to the user's vision field at differentamounts of at least one of divergence and collimation and thus thedisplayed virtual image content appears to originate from differentdepths.

Example 115: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content to the user's vision field such that thedisplayed virtual image content appears to originate from differentdepths.

Example 116: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content to the user's vision field at differentamounts of divergence such that the displayed virtual image contentappears to originate from different depths.

Example 117: The display system of any of the Examples above, whereinsaid display is configured to project light into said user's eye thatdivergences and to project light into said user's eye that is collimatedto display virtual image content to the user's vision field that appearsto originate from different depths.

Example 118: The display system of any of the Examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 119: The display system of any of the Examples above, whereinsaid first parameter comprises a center of rotation of the eye.

Example 120: The display system of any of the Examples above, whereinsaid second parameter comprises a center of curvature of the cornea.

Example 121: The display system of any of the Examples above, whereinsaid parameter comprises a center of curvature of the cornea.

Example 122: The display system of any of the Examples above, whereinsaid additional parameter comprises a center of rotation of the eye.

Any of the above Examples can be combined. Additionally, any of theabove Examples can be integrated with a head mounted display. Inaddition, any of the above Examples can be implemented with a singledepth plane and/or with one or more depth planes such as one or morevariable depth planes (e.g., one or more elements with variable focusingpower that provide accommodation cues that vary over time).

Furthermore, apparatus and methods for determining a variety of values,parameters, etc., such as, but not limited to, anatomical, optical, andgeometric features, locations, and orientations, etc., are disclosedherein. Examples of such parameters include, for example, the center ofrotation of the eye, the center of curvature of the cornea, the centerof the pupil, the boundary of the pupil, the center of the iris, theboundary of the iris, the boundary of the limbus, the optical axis ofthe eye, the visual axis of the eye, the center of perspective, but arenot limited to these. Additionally, in some implementations, the centerof curvature of the cornea or the center of the cornea refers to thecenter of curvature of a portion of the cornea or the center ofcurvature of a spherical surface that coincides with a portion of thesurface of the cornea. For example, in some implementations, the centerof curvature of the cornea or the center of the cornea refers to thecenter of curvature of the cornea apex or the center of curvature of aspherical surface that coincides with a portion of the surface of thecorneal apex. In addition, determinations of such values, parameters,etc., as recited herein include estimations thereof and need notnecessarily coincide precisely with the actual values. For example,determinations of the center of rotation of the eye, the center ofcurvature of the cornea, the center or boundary of the pupil or iris,the boundary of the limbus, the optical axis of the eye, the visual axisof the eye, the center of perspective, etc., may be estimations,approximations, or values close to, but not the same as, the actual(e.g., anatomical, optical, or geometric) values or parameters. In somecases, for example, root mean square estimation techniques are used toobtain estimates of such values. As an example, certain techniquesdescribed herein relate to identifying a location or point at which raysor vectors intersect. Such rays or vectors, however, may not intersect.In this example, the location or point may be estimated. For example,the location or point may be determined based on root mean square, orother, estimation techniques (e.g., the location or point may beestimated to be close to or the closest to the rays or vectors). Otherprocesses may also be used to estimate, approximate or otherwise providea value that may not coincide with the actual value. Accordingly, theterm determining and estimating, or determined and estimated, are usedinterchangeably herein. Reference to such determined values maytherefore include estimates, approximations, or values close to theactual value. Accordingly, reference to determining a parameter or valueabove, or elsewhere herein should not be limited precisely to the actualvalue but may include estimations, approximations or values closethereto.

Details of one or more implementations of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages will becomeapparent from the description, the drawings, and the claims. Neitherthis summary nor the following detailed description purports to defineor limit the scope of the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson.

FIG. 2 schematically illustrates an example of a wearable system.

FIG. 3 schematically illustrates example components of a wearablesystem.

FIG. 4 schematically illustrates an example of a waveguide stack of awearable device for outputting image information to a user.

FIG. 5 schematically illustrates an example of an eye.

FIG. 5A schematically illustrates an example coordinate system fordetermining an eye pose of an eye

FIG. 6 is a schematic diagram of a wearable system that includes an eyetracking system.

FIG. 7A is a block diagram of a wearable system that may include an eyetracking system.

FIG. 7B is a block diagram of a render controller in a wearable system.

FIG. 8 illustrates an example of an eye including the eye's optical andvisual axes and the eye's center of rotation.

FIGS. 9A-E illustrate views of an example configuration of a wearablesystem for capturing eye image data for use by an eye tracking module.

FIG. 10 shows a graphical illustration of an example Center of Rotation(CoR) determination process that may be performed by an eye trackingmodule.

FIG. 11 shows example images of glints on any eye used by the eyetracking module for determining an estimated center of rotation.

FIGS. 12A-D illustrate steps in an example determination of a firstplane that includes the locations of a first glint, the camera capturingthe image of the glint, and the source of illumination producing thefirst glint.

FIGS. 13A-D illustrate steps in an example determination of a secondplane that includes the locations of a second glint, the camera, and thesource of illumination producing the second glint.

FIGS. 14A-C illustrate the intersection between the first plane of FIGS.12A-D and the second plane of FIGS. 13A-D. This intersection correspondsto a vector along which the cornea center may lie.

FIGS. 15A-15B illustrate a plurality of vector along which the corneacenter may lie obtained using multiple cameras. These vectors mayconverge or intersect at a location corresponding to or close to thecorneal center.

FIGS. 16A-16C illustrate example steps in an example determination of avectors along which the cornea center may lie using shared illuminationsources between multiple cameras.

FIGS. 17A-17B show estimation of a 3D surface based on calculated corneacenters.

FIGS. 18A and 18B show an example estimation of center of rotation atthe convergence of a plurality of surface normal vectors normal to the3D surface calculated based on the corneal centers.

FIGS. 19A-1 and 19A-2 illustrate an example surface fit to selectedestimated cornea centers.

FIGS. 19B-1 and 19B-2 shows example surface normal vectors that may benormal to a surface fitted to estimated cornea centers.

FIGS. 19C-1 and 19C-2 illustrate an estimated CoR region based on pointsof intersection of the surface normal vectors.

FIGS. 19D-1 and 19D-2, illustrate an example surface fit to anotherselection of cornea centers.

FIG. 20 illustrates an example center of rotation extraction processthat may be implemented by an eye tracking module.

FIG. 21 illustrates an example eye tracking process that may use theprocess of FIG. 20 for determining estimated of centers of rotationusing centers of corneal curvature.

Throughout the drawings, reference numbers may be re-used to indicatecorrespondence between referenced elements. The drawings are provided toillustrate example embodiments described herein and are not intended tolimit the scope of the disclosure.

DETAILED DESCRIPTION

Reference will now be made to the drawings, in which like referencenumerals refer to like parts throughout. Unless indicated otherwise, thedrawings are schematic not necessarily drawn to scale.

A. EXAMPLES OF 3D DISPLAY OF A WEARABLE SYSTEM

A wearable system (also referred to herein as an augmented reality (AR)system) can be configured to present 2D or 3D virtual images to a user.The images may be still images, frames of a video, or a video, incombination or the like. At least a portion of the wearable system canbe implemented on a wearable device that can present a VR, AR, or MRenvironment, alone or in combination, for user interaction. The wearabledevice can be used interchangeably as an AR device (ARD). Further, forthe purpose of the present disclosure, the term “AR” is usedinterchangeably with the term “MR”.

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson. In FIG. 1, an MR scene 100 is depicted wherein a user of an MRtechnology sees a real-world park-like setting 110 featuring people,trees, buildings in the background, and a concrete platform 120. Inaddition to these items, the user of the MR technology also perceivesthat he “sees” a robot statue 130 standing upon the real-world platform120, and a cartoon-like avatar character 140 flying by which seems to bea personification of a bumble bee, even though these elements do notexist in the real world.

In order for the 3D display to produce a true sensation of depth, andmore specifically, a simulated sensation of surface depth, it may bedesirable for each point in the display's visual field to generate anaccommodative response corresponding to its virtual depth. If theaccommodative response to a display point does not correspond to thevirtual depth of that point, as determined by the binocular depth cuesof convergence and stereopsis, the human eye may experience anaccommodation conflict, resulting in unstable imaging, harmful eyestrain, headaches, and, in the absence of accommodation information,almost a complete lack of surface depth.

VR, AR, and MR experiences can be provided by display systems havingdisplays in which images corresponding to a plurality of depth planesare provided to a viewer. The images may be different for each depthplane (e.g., provide slightly different presentations of a scene orobject) and may be separately focused by the viewer's eyes, therebyhelping to provide the user with depth cues based on the accommodationof the eye required to bring into focus different image features for thescene located on different depth plane or based on observing differentimage features on different depth planes being out of focus. Asdiscussed elsewhere herein, such depth cues provide credible perceptionsof depth.

FIG. 2 illustrates an example of wearable system 200 which can beconfigured to provide an AR/VR/MR scene. The wearable system 200 canalso be referred to as the AR system 200. The wearable system 200includes a display 220, and various mechanical and electronic modulesand systems to support the functioning of display 220. The display 220may be coupled to a frame 230, which is wearable by a user, wearer, orviewer 210. The display 220 can be positioned in front of the eyes ofthe user 210. The display 220 can present AR/VR/MR content to a user.The display 220 can comprise a head mounted display (HMD) that is wornon the head of the user.

In some embodiments, a speaker 240 is coupled to the frame 230 andpositioned adjacent the ear canal of the user (in some embodiments,another speaker, not shown, is positioned adjacent the other ear canalof the user to provide for stereo/shapeable sound control). The display220 can include an audio sensor (e.g., a microphone) 232 for detectingan audio stream from the environment and capture ambient sound. In someembodiments, one or more other audio sensors, not shown, are positionedto provide stereo sound reception. Stereo sound reception can be used todetermine the location of a sound source. The wearable system 200 canperform voice or speech recognition on the audio stream.

The wearable system 200 can include an outward-facing imaging system 464(shown in FIG. 4) which observes the world in the environment around theuser. The wearable system 200 can also include an inward-facing imagingsystem 462 (shown in FIG. 4) which can track the eye movements of theuser. The inward-facing imaging system may track either one eye'smovements or both eyes' movements. The inward-facing imaging system 462may be attached to the frame 230 and may be in electrical communicationwith the processing modules 260 or 270, which may process imageinformation acquired by the inward-facing imaging system to determine,e.g., the pupil diameters or orientations of the eyes, eye movements oreye pose of the user 210. The inward-facing imaging system 462 mayinclude one or more cameras. For example, at least one camera may beused to image each eye. The images acquired by the cameras may be usedto determine pupil size or eye pose for each eye separately, therebyallowing presentation of image information to each eye to be dynamicallytailored to that eye.

As an example, the wearable system 200 can use the outward-facingimaging system 464 or the inward-facing imaging system 462 to acquireimages of a pose of the user. The images may be still images, frames ofa video, or a video.

The display 220 can be operatively coupled 250, such as by a wired leador wireless connectivity, to a local data processing module 260 whichmay be mounted in a variety of configurations, such as fixedly attachedto the frame 230, fixedly attached to a helmet or hat worn by the user,embedded in headphones, or otherwise removably attached to the user 210(e.g., in a backpack-style configuration, in a belt-coupling styleconfiguration).

The local processing and data module 260 may comprise a hardwareprocessor, as well as digital memory, such as non-volatile memory (e.g.,flash memory), both of which may be utilized to assist in theprocessing, caching, and storage of data. The data may include data a)captured from sensors (which may be, e.g., operatively coupled to theframe 230 or otherwise attached to the user 210), such as image capturedevices (e.g., cameras in the inward-facing imaging system or theoutward-facing imaging system), audio sensors (e.g., microphones),inertial measurement units (IMUs), accelerometers, compasses, globalpositioning system (GPS) units, radio devices, or gyroscopes; or b)acquired or processed using remote processing module 270 or remote datarepository 280, possibly for passage to the display 220 after suchprocessing or retrieval. The local processing and data module 260 may beoperatively coupled by communication links 262 or 264, such as via wiredor wireless communication links, to the remote processing module 270 orremote data repository 280 such that these remote modules are availableas resources to the local processing and data module 260. In addition,remote processing module 280 and remote data repository 280 may beoperatively coupled to each other.

In some embodiments, the remote processing module 270 may comprise oneor more processors configured to analyze and process data or imageinformation. In some embodiments, the remote data repository 280 maycomprise a digital data storage facility, which may be available throughthe internet or other networking configuration in a “cloud” resourceconfiguration. In some embodiments, all data is stored and allcomputations are performed in the local processing and data module,allowing fully autonomous use from a remote module.

B. EXAMPLE COMPONENTS OF A WEARABLE SYSTEM

FIG. 3 schematically illustrates example components of a wearablesystem. FIG. 3 shows a wearable system 200 which can include a display220 and a frame 230. A blown-up view 202 schematically illustratesvarious components of the wearable system 200. In certain implements,one or more of the components illustrated in FIG. 3 can be part of thedisplay 220. The various components alone or in combination can collecta variety of data (such as e.g., audio or visual data) associated withthe user of the wearable system 200 or the user's environment. It shouldbe appreciated that other embodiments may have additional or fewercomponents depending on the application for which the wearable system isused. Nevertheless, FIG. 3 provides a basic idea of some of the variouscomponents and types of data that may be collected, analyzed, and storedthrough the wearable system.

FIG. 3 shows an example wearable system 200 which can include thedisplay 220. The display 220 can comprise a display lens 226 that may bemounted to a user's head or a housing or frame 230, which corresponds tothe frame 230. The display lens 226 may comprise one or more transparentmirrors positioned by the housing 230 in front of the user's eyes 302,304 and may be configured to bounce projected light 338 into the eyes302, 304 and facilitate beam shaping, while also allowing fortransmission of at least some light from the local environment. Thewavefront of the projected light beam 338 may be bent or focused tocoincide with a desired focal distance of the projected light. Asillustrated, two wide-field-of-view machine vision cameras 316 (alsoreferred to as world cameras) can be coupled to the housing 230 to imagethe environment around the user. These cameras 316 can be dual capturevisible light/non-visible (e.g., infrared) light cameras. The cameras316 may be part of the outward-facing imaging system 464 shown in FIG.4. Image acquired by the world cameras 316 can be processed by the poseprocessor 336. For example, the pose processor 336 can implement one ormore object recognizers 708 to identify a pose of a user or anotherperson in the user's environment or to identify a physical object in theuser's environment.

With continued reference to FIG. 3, a pair of scanned-lasershaped-wavefront (e.g., for depth) light projector modules with displaymirrors and optics configured to project light 338 into the eyes 302,304 are shown. The depicted view also shows two miniature infraredcameras 324 paired with infrared light sources 326 (such as lightemitting diodes “LED”s), which are configured to be able to track theeyes 302, 304 of the user to support rendering and user input. Thecameras 324 may be part of the inward-facing imaging system 462 shown inFIG. 4. The wearable system 200 can further feature a sensor assembly339, which may comprise X, Y, and Z axis accelerometer capability aswell as a magnetic compass and X, Y, and Z axis gyro capability,preferably providing data at a relatively high frequency, such as 200Hz. The sensor assembly 339 may be part of the IMU described withreference to FIG. 2 The depicted system 200 can also comprise a headpose processor 336, such as an ASIC (application specific integratedcircuit), FPGA (field programmable gate array), or ARM processor(advanced reduced-instruction-set machine), which may be configured tocalculate real or near-real time user head pose from wide field of viewimage information output from the capture devices 316. The head poseprocessor 336 can be a hardware processor and can be implemented as partof the local processing and data module 260 shown in FIG. 2.

The wearable system can also include one or more depth sensors 234. Thedepth sensor 234 can be configured to measure the distance between anobject in an environment to a wearable device. The depth sensor 234 mayinclude a laser scanner (e.g., a lidar), an ultrasonic depth sensor, ora depth sensing camera. In certain implementations, where the cameras316 have depth sensing ability, the cameras 316 may also be consideredas depth sensors 234.

Also shown is a processor 332 configured to execute digital or analogprocessing to derive pose from the gyro, compass, or accelerometer datafrom the sensor assembly 339. The processor 332 may be part of the localprocessing and data module 260 shown in FIG. 2. The wearable system 200as shown in FIG. 3 can also include a position system such as, e.g., aGPS 337 (global positioning system) to assist with pose and positioninganalyses. In addition, the GPS may further provide remotely-based (e.g.,cloud-based) information about the user's environment. This informationmay be used for recognizing objects or information in user'senvironment.

The wearable system may combine data acquired by the GPS 337 and aremote computing system (such as, e.g., the remote processing module270, another user's ARD, etc.) which can provide more information aboutthe user's environment. As one example, the wearable system candetermine the user's location based on GPS data and retrieve a world map(e.g., by communicating with a remote processing module 270) includingvirtual objects associated with the user's location. As another example,the wearable system 200 can monitor the environment using the worldcameras 316 (which may be part of the outward-facing imaging system 464shown in FIG. 4). Based on the images acquired by the world cameras 316,the wearable system 200 can detect objects in the environment (e.g., byusing one or more object recognizers). The wearable system can furtheruse data acquired by the GPS 337 to interpret the characters.

The wearable system 200 may also comprise a rendering engine 334 whichcan be configured to provide rendering information that is local to theuser to facilitate operation of the scanners and imaging into the eyesof the user, for the user's view of the world. The rendering engine 334may be implemented by a hardware processor (such as, e.g., a centralprocessing unit or a graphics processing unit). In some embodiments, therendering engine is part of the local processing and data module 260.The rendering engine 334 can be communicatively coupled (e.g., via wiredor wireless links) to other components of the wearable system 200. Forexample, the rendering engine 334, can be coupled to the eye cameras 324via communication link 274, and be coupled to a projecting subsystem 318(which can project light into user's eyes 302, 304 via a scanned laserarrangement in a manner similar to a retinal scanning display) via thecommunication link 272. The rendering engine 334 can also be incommunication with other processing units such as, e.g., the sensor poseprocessor 332 and the image pose processor 336 via links 276 and 294respectively.

The cameras 324 (e.g., mini infrared cameras) may be utilized to trackthe eye pose to support rendering and user input. Some example eye posesmay include where the user is looking or at what depth he or she isfocusing (which may be estimated with eye vergence). The GPS 337, gyros,compass, and accelerometers 339 may be utilized to provide coarse orfast pose estimates. One or more of the cameras 316 can acquire imagesand pose, which in conjunction with data from an associated cloudcomputing resource, may be utilized to map the local environment andshare user views with others.

The example components depicted in FIG. 3 are for illustration purposesonly. Multiple sensors and other functional modules are shown togetherfor ease of illustration and description. Some embodiments may includeonly one or a subset of these sensors or modules. Further, the locationsof these components are not limited to the positions depicted in FIG. 3.Some components may be mounted to or housed within other components,such as a belt-mounted component, a hand-held component, or a helmetcomponent. As one example, the image pose processor 336, sensor poseprocessor 332, and rendering engine 334 may be positioned in a beltpackand configured to communicate with other components of the wearablesystem via wireless communication, such as ultra-wideband, Wi-Fi,Bluetooth, etc., or via wired communication. The depicted housing 230preferably is head-mountable and wearable by the user. However, somecomponents of the wearable system 200 may be worn to other portions ofthe user's body. For example, the speaker 240 may be inserted into theears of a user to provide sound to the user.

Regarding the projection of light 338 into the eyes 302, 304 of theuser, in some embodiment, the cameras 324 may be utilized to measurewhere the centers of a user's eyes are geometrically verged to, which,in general, coincides with a position of focus, or “depth of focus”, ofthe eyes. A 3-dimensional surface of all points the eyes verge to can bereferred to as the “horopter”. The focal distance may take on a finitenumber of depths, or may be infinitely varying. Light projected from thevergence distance appears to be focused to the subject eye 302, 304,while light in front of or behind the vergence distance is blurred.Examples of wearable devices and other display systems of the presentdisclosure are also described in U.S. Patent Publication No.2016/0270656, which is incorporated by reference herein in its entirety.

The human visual system is complicated and providing a realisticperception of depth is challenging. Viewers of an object may perceivethe object as being three-dimensional due to a combination of vergenceand accommodation. Vergence movements (e.g., rolling movements of thepupils toward or away from each other to converge the lines of sight ofthe eyes to fixate upon an object) of the two eyes relative to eachother are closely associated with focusing (or “accommodation”) of thelenses of the eyes. Under normal conditions, changing the focus of thelenses of the eyes, or accommodating the eyes, to change focus from oneobject to another object at a different distance will automaticallycause a matching change in vergence to the same distance, under arelationship known as the “accommodation-vergence reflex.” Likewise, achange in vergence will trigger a matching change in accommodation,under normal conditions. Display systems that provide a better matchbetween accommodation and vergence may form more realistic andcomfortable simulations of three-dimensional imagery.

Further spatially coherent light with a beam diameter of less than about0.7 millimeters can be correctly resolved by the human eye regardless ofwhere the eye focuses. Thus, to create an illusion of proper focaldepth, the eye vergence may be tracked with the cameras 324, and therendering engine 334 and projection subsystem 318 may be utilized torender all objects on or close to the horopter in focus, and all otherobjects at varying degrees of defocus (e.g., using intentionally-createdblurring). Preferably, the system 220 renders to the user at a framerate of about 60 frames per second or greater. As described above,preferably, the cameras 324 may be utilized for eye tracking, andsoftware may be configured to pick up not only vergence geometry butalso focus location cues to serve as user inputs. Preferably, such adisplay system is configured with brightness and contrast suitable forday or night use.

In some embodiments, the display system preferably has latency of lessthan about 20 milliseconds for visual object alignment, less than about0.1 degree of angular alignment, and about 1 arc minute of resolution,which, without being limited by theory, is believed to be approximatelythe limit of the human eye. The display system 220 may be integratedwith a localization system, which may involve GPS elements, opticaltracking, compass, accelerometers, or other data sources, to assist withposition and pose determination; localization information may beutilized to facilitate accurate rendering in the user's view of thepertinent world (e.g., such information would facilitate the glasses toknow where they are with respect to the real world).

In some embodiments, the wearable system 200 is configured to displayone or more virtual images based on the accommodation of the user'seyes. Unlike prior 3D display approaches that force the user to focuswhere the images are being projected, in some embodiments, the wearablesystem is configured to automatically vary the focus of projectedvirtual content to allow for a more comfortable viewing of one or moreimages presented to the user. For example, if the user's eyes have acurrent focus of 1 m, the image may be projected to coincide with theuser's focus. If the user shifts focus to 3 m, the image is projected tocoincide with the new focus. Thus, rather than forcing the user to apredetermined focus, the wearable system 200 of some embodiments allowsthe user's eye to a function in a more natural manner.

Such a wearable system 200 may eliminate or reduce the incidences of eyestrain, headaches, and other physiological symptoms typically observedwith respect to virtual reality devices. To achieve this, variousembodiments of the wearable system 200 are configured to project virtualimages at varying focal distances, through one or more variable focuselements (VFEs). In one or more embodiments, 3D perception may beachieved through a multi-plane focus system that projects images atfixed focal planes away from the user. Other embodiments employ variableplane focus, wherein the focal plane is moved back and forth in thez-direction to coincide with the user's present state of focus.

In both the multi-plane focus systems and variable plane focus systems,wearable system 200 may employ eye tracking to determine a vergence ofthe user's eyes, determine the user's current focus, and project thevirtual image at the determined focus. In other embodiments, wearablesystem 200 comprises a light modulator that variably projects, through afiber scanner, or other light generating source, light beams of varyingfocus in a raster pattern across the retina. Thus, the ability of thedisplay of the wearable system 200 to project images at varying focaldistances not only eases accommodation for the user to view objects in3D, but may also be used to compensate for user ocular anomalies, asfurther described in U.S. Patent Publication No. 2016/0270656, which isincorporated by reference herein in its entirety. In some otherembodiments, a spatial light modulator may project the images to theuser through various optical components. For example, as describedfurther below, the spatial light modulator may project the images ontoone or more waveguides, which then transmit the images to the user.

C. WAVEGUIDE STACK ASSEMBLY

FIG. 4 illustrates an example of a waveguide stack for outputting imageinformation to a user. A wearable system 400 includes a stack ofwaveguides, or stacked waveguide assembly 480 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 432 b, 434 b, 436 b, 438 b, 4400 b. In some embodiments,the wearable system 400 may correspond to wearable system 200 of FIG. 2,with FIG. 4 schematically showing some parts of that wearable system 200in greater detail. For example, in some embodiments, the waveguideassembly 480 may be integrated into the display 220 of FIG. 2.

With continued reference to FIG. 4, the waveguide assembly 480 may alsoinclude a plurality of features 458, 456, 454, 452 between thewaveguides. In some embodiments, the features 458, 456, 454, 452 may belenses. In other embodiments, the features 458, 456, 454, 452 may not belenses. Rather, they may simply be spacers (e.g., cladding layers orstructures for forming air gaps).

The waveguides 432 b, 434 b, 436 b, 438 b, 440 b or the plurality oflenses 458, 456, 454, 452 may be configured to send image information tothe eye with various levels of wavefront curvature or light raydivergence. Each waveguide level may be associated with a particulardepth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 420, 422,424, 426, 428 may be utilized to inject image information into thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b, each of which may beconfigured to distribute incoming light across each respectivewaveguide, for output toward the eye 410. Light exits an output surfaceof the image injection devices 420, 422, 424, 426, 428 and is injectedinto a corresponding input edge of the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, a single beam of light (e.g., acollimated beam) may be injected into each waveguide to output an entirefield of cloned collimated beams that are directed toward the eye 410 atparticular angles (and amounts of divergence) corresponding to the depthplane associated with a particular waveguide.

In some embodiments, the image injection devices 420, 422, 424, 426, 428are discrete displays that each produce image information for injectioninto a corresponding waveguide 440 b, 438 b, 436 b, 434 b, 432 b,respectively. In some other embodiments, the image injection devices420, 422, 424, 426, 428 are the output ends of a single multiplexeddisplay which may, e.g., pipe image information via one or more opticalconduits (such as fiber optic cables) to each of the image injectiondevices 420, 422, 424, 426, 428.

A controller 460 controls the operation of the stacked waveguideassembly 480 and the image injection devices 420, 422, 424, 426, 428.The controller 460 includes programming (e.g., instructions in anon-transitory computer-readable medium) that regulates the timing andprovision of image information to the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, the controller 460 may be a singleintegral device, or a distributed system connected by wired or wirelesscommunication channels. The controller 460 may be part of the processingmodules 260 or 270 (illustrated in FIG. 2) in some embodiments.

The waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be configured topropagate light within each respective waveguide by total internalreflection (TIR). The waveguides 440 b, 438 b, 436 b, 434 b, 432 b mayeach be planar or have another shape (e.g., curved), with major top andbottom surfaces and edges extending between those major top and bottomsurfaces. In the illustrated configuration, the waveguides 440 b, 438 b,436 b, 434 b, 432 b may each include light extracting optical elements440 a, 438 a, 436 a, 434 a, 432 a that are configured to extract lightout of a waveguide by redirecting the light, propagating within eachrespective waveguide, out of the waveguide to output image informationto the eye 410. Extracted light may also be referred to as outcoupledlight, and light extracting optical elements may also be referred to asoutcoupling optical elements. An extracted beam of light is outputted bythe waveguide at locations at which the light propagating in thewaveguide strikes a light redirecting element. The light extractingoptical elements (440 a, 438 a, 436 a, 434 a, 432 a) may, for example,be reflective or diffractive optical features. While illustrateddisposed at the bottom major surfaces of the waveguides 440 b, 438 b,436 b, 434 b, 432 b for ease of description and drawing clarity, in someembodiments, the light extracting optical elements 440 a, 438 a, 436 a,434 a, 432 a may be disposed at the top or bottom major surfaces, or maybe disposed directly in the volume of the waveguides 440 b, 438 b, 436b, 434 b, 432 b. In some embodiments, the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be formed in a layer ofmaterial that is attached to a transparent substrate to form thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b. In some other embodiments,the waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be a monolithicpiece of material and the light extracting optical elements 440 a, 438a, 436 a, 434 a, 432 a may be formed on a surface or in the interior ofthat piece of material.

With continued reference to FIG. 4, as discussed herein, each waveguide440 b, 438 b, 436 b, 434 b, 432 b is configured to output light to forman image corresponding to a particular depth plane. For example, thewaveguide 432 b nearest the eye may be configured to deliver collimatedlight, as injected into such waveguide 432 b, to the eye 410. Thecollimated light may be representative of the optical infinity focalplane. The next waveguide up 434 b may be configured to send outcollimated light which passes through the first lens 452 (e.g., anegative lens) before it can reach the eye 410. First lens 452 may beconfigured to create a slight convex wavefront curvature so that theeye/brain interprets light coming from that next waveguide up 434 b ascoming from a first focal plane closer inward toward the eye 410 fromoptical infinity. Similarly, the third up waveguide 436 b passes itsoutput light through both the first lens 452 and second lens 454 beforereaching the eye 410. The combined optical power of the first and secondlenses 452 and 454 may be configured to create another incrementalamount of wavefront curvature so that the eye/brain interprets lightcoming from the third waveguide 436 b as coming from a second focalplane that is even closer inward toward the person from optical infinitythan was light from the next waveguide up 434 b.

The other waveguide layers (e.g., waveguides 438 b, 440 b) and lenses(e.g., lenses 456, 458) are similarly configured, with the highestwaveguide 440 b in the stack sending its output through all of thelenses between it and the eye for an aggregate focal powerrepresentative of the closest focal plane to the person. To compensatefor the stack of lenses 458, 456, 454, 452 when viewing/interpretinglight coming from the world 470 on the other side of the stackedwaveguide assembly 480, a compensating lens layer 430 may be disposed atthe top of the stack to compensate for the aggregate power of the lensstack 458, 456, 454, 452 below. (Compensating lens layer 430 and thestacked waveguide assembly 480 as a whole may be configured such thatlight coming from the world 470 is conveyed to the eye 410 atsubstantially the same level of divergence (or collimation) as the lighthad when it was initially received by the stacked waveguide assembly480.) Such a configuration provides as many perceived focal planes asthere are available waveguide/lens pairings. Both the light extractingoptical elements of the waveguides and the focusing aspects of thelenses may be static (e.g., not dynamic or electro-active). In somealternative embodiments, either or both may be dynamic usingelectro-active features.

With continued reference to FIG. 4, the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be configured to bothredirect light out of their respective waveguides and to output thislight with the appropriate amount of divergence or collimation for aparticular depth plane associated with the waveguide. As a result,waveguides having different associated depth planes may have differentconfigurations of light extracting optical elements, which output lightwith a different amount of divergence depending on the associated depthplane. In some embodiments, as discussed herein, the light extractingoptical elements 440 a, 438 a, 436 a, 434 a, 432 a may be volumetric orsurface features, which may be configured to output light at specificangles. For example, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a may be volume holograms, surface holograms, and/ordiffraction gratings. Light extracting optical elements, such asdiffraction gratings, are described in U.S. Patent Publication No.2015/0178939, published Jun. 25, 2015, which is incorporated byreference herein in its entirety.

In some embodiments, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a are diffractive features that form a diffractionpattern, or “diffractive optical element” (also referred to herein as a“DOE”). Preferably, the DOE has a relatively low diffraction efficiencyso that only a portion of the light of the beam is deflected away towardthe eye 410 with each intersection of the DOE, while the rest continuesto move through a waveguide via total internal reflection. The lightcarrying the image information can thus be divided into a number ofrelated exit beams that exit the waveguide at a multiplicity oflocations and the result is a fairly uniform pattern of exit emissiontoward the eye 304 for this particular collimated beam bouncing aroundwithin a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”state in which they actively diffract, and “off” state in which they donot significantly diffract. For instance, a switchable DOE may comprisea layer of polymer dispersed liquid crystal, in which microdropletscomprise a diffraction pattern in a host medium, and the refractiveindex of the microdroplets can be switched to substantially match therefractive index of the host material (in which case the pattern doesnot appreciably diffract incident light) or the microdroplet can beswitched to an index that does not match that of the host medium (inwhich case the pattern actively diffracts incident light).

In some embodiments, the number and distribution of depth planes ordepth of field may be varied dynamically based on the pupil sizes ororientations of the eyes of the viewer. Depth of field may changeinversely with a viewer's pupil size. As a result, as the sizes of thepupils of the viewer's eyes decrease, the depth of field increases suchthat one plane that is not discernible because the location of thatplane is beyond the depth of focus of the eye may become discernible andappear more in focus with reduction of pupil size and commensurate withthe increase in depth of field. Likewise, the number of spaced apartdepth planes used to present different images to the viewer may bedecreased with the decreased pupil size. For example, a viewer may notbe able to clearly perceive the details of both a first depth plane anda second depth plane at one pupil size without adjusting theaccommodation of the eye away from one depth plane and to the otherdepth plane. These two depth planes may, however, be sufficiently infocus at the same time to the user at another pupil size withoutchanging accommodation.

In some embodiments, the display system may vary the number ofwaveguides receiving image information based upon determinations ofpupil size or orientation, or upon receiving electrical signalsindicative of particular pupil size or orientation. For example, if theuser's eyes are unable to distinguish between two depth planesassociated with two waveguides, then the controller 460 (which may be anembodiment of the local processing and data module 260) can beconfigured or programmed to cease providing image information to one ofthese waveguides. Advantageously, this may reduce the processing burdenon the system, thereby increasing the responsiveness of the system. Inembodiments in which the DOEs for a waveguide are switchable between theon and off states, the DOEs may be switched to the off state when thewaveguide does receive image information.

In some embodiments, it may be desirable to have an exit beam meet thecondition of having a diameter that is less than the diameter of the eyeof a viewer. However, meeting this condition may be challenging in viewof the variability in size of the viewer's pupils. In some embodiments,this condition is met over a wide range of pupil sizes by varying thesize of the exit beam in response to determinations of the size of theviewer's pupil. For example, as the pupil size decreases, the size ofthe exit beam may also decrease. In some embodiments, the exit beam sizemay be varied using a variable aperture.

The wearable system 400 can include an outward-facing imaging system 464(e.g., a digital camera) that images a portion of the world 470. Thisportion of the world 470 may be referred to as the field of view (FOV)of a world camera and the imaging system 464 is sometimes referred to asan FOV camera. The FOV of the world camera may or may not be the same asthe FOV of a viewer 210 which encompasses a portion of the world 470 theviewer 210 perceives at a given time. For example, in some situations,the FOV of the world camera may be larger than the viewer 210 of theviewer 210 of the wearable system 400. The entire region available forviewing or imaging by a viewer may be referred to as the field of regard(FOR). The FOR may include 4π steradians of solid angle surrounding thewearable system 400 because the wearer can move his body, head, or eyesto perceive substantially any direction in space. In other contexts, thewearer's movements may be more constricted, and accordingly the wearer'sFOR may subtend a smaller solid angle. Images obtained from theoutward-facing imaging system 464 can be used to track gestures made bythe user (e.g., hand or finger gestures), detect objects in the world470 in front of the user, and so forth.

The wearable system 400 can include an audio sensor 232, e.g., amicrophone, to capture ambient sound. As described above, in someembodiments, one or more other audio sensors can be positioned toprovide stereo sound reception useful to the determination of locationof a speech source. The audio sensor 232 can comprise a directionalmicrophone, as another example, which can also provide such usefuldirectional information as to where the audio source is located. Thewearable system 400 can use information from both the outward-facingimaging system 464 and the audio sensor 230 in locating a source ofspeech, or to determine an active speaker at a particular moment intime, etc. For example, the wearable system 400 can use the voicerecognition alone or in combination with a reflected image of thespeaker (e.g., as seen in a mirror) to determine the identity of thespeaker. As another example, the wearable system 400 can determine aposition of the speaker in an environment based on sound acquired fromdirectional microphones. The wearable system 400 can parse the soundcoming from the speaker's position with speech recognition algorithms todetermine the content of the speech and use voice recognition techniquesto determine the identity (e.g., name or other demographic information)of the speaker.

The wearable system 400 can also include an inward-facing imaging system466 (e.g., a digital camera), which observes the movements of the user,such as the eye movements and the facial movements. The inward-facingimaging system 466 may be used to capture images of the eye 410 todetermine the size and/or orientation of the pupil of the eye 304. Theinward-facing imaging system 466 can be used to obtain images for use indetermining the direction the user is looking (e.g., eye pose) or forbiometric identification of the user (e.g., via iris identification). Insome embodiments, at least one camera may be utilized for each eye, toseparately determine the pupil size or eye pose of each eyeindependently, thereby allowing the presentation of image information toeach eye to be dynamically tailored to that eye. In some otherembodiments, the pupil diameter or orientation of only a single eye 410(e.g., using only a single camera per pair of eyes) is determined andassumed to be similar for both eyes of the user. The images obtained bythe inward-facing imaging system 466 may be analyzed to determine theuser's eye pose or mood, which can be used by the wearable system 400 todecide which audio or visual content should be presented to the user.The wearable system 400 may also determine head pose (e.g., headposition or head orientation) using sensors such as IMUs,accelerometers, gyroscopes, etc.

The wearable system 400 can include a user input device 466 by which theuser can input commands to the controller 460 to interact with thewearable system 400. For example, the user input device 466 can includea trackpad, a touchscreen, a joystick, a multiple degree-of-freedom(DOF) controller, a capacitive sensing device, a game controller, akeyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, atotem (e.g., functioning as a virtual user input device), and so forth.A multi-DOF controller can sense user input in some or all possibletranslations (e.g., left/right, forward/backward, or up/down) orrotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOFcontroller which supports the translation movements may be referred toas a 3DOF while a multi-DOF controller which supports the translationsand rotations may be referred to as 6DOF. In some cases, the user mayuse a finger (e.g., a thumb) to press or swipe on a touch-sensitiveinput device to provide input to the wearable system 400 (e.g., toprovide user input to a user interface provided by the wearable system400). The user input device 466 may be held by the user's hand duringthe use of the wearable system 400. The user input device 466 can be inwired or wireless communication with the wearable system 400.

D. OTHER COMPONENTS OF THE WEARABLE SYSTEM

In many implementations, the wearable system may include othercomponents in addition or in alternative to the components of thewearable system described above. The wearable system may, for example,include one or more haptic devices or components. The haptic devices orcomponents may be operable to provide a tactile sensation to a user. Forexample, the haptic devices or components may provide a tactilesensation of pressure or texture when touching virtual content (e.g.,virtual objects, virtual tools, other virtual constructs). The tactilesensation may replicate a feel of a physical object which a virtualobject represents, or may replicate a feel of an imagined object orcharacter (e.g., a dragon) which the virtual content represents. In someimplementations, haptic devices or components may be worn by the user(e.g., a user wearable glove). In some implementations, haptic devicesor components may be held by the user.

The wearable system may, for example, include one or more physicalobjects which are manipulable by the user to allow input or interactionwith the wearable system. These physical objects may be referred toherein as totems. Some totems may take the form of inanimate objects,such as for example, a piece of metal or plastic, a wall, a surface oftable. In certain implementations, the totems may not actually have anyphysical input structures (e.g., keys, triggers, joystick, trackball,rocker switch). Instead, the totem may simply provide a physicalsurface, and the wearable system may render a user interface so as toappear to a user to be on one or more surfaces of the totem. Forexample, the wearable system may render an image of a computer keyboardand trackpad to appear to reside on one or more surfaces of a totem. Forexample, the wearable system may render a virtual computer keyboard andvirtual trackpad to appear on a surface of a thin rectangular plate ofaluminum which serves as a totem. The rectangular plate does not itselfhave any physical keys or trackpad or sensors. However, the wearablesystem may detect user manipulation or interaction or touches with therectangular plate as selections or inputs made via the virtual keyboardor virtual trackpad. The user input device 466 (shown in FIG. 4) may bean embodiment of a totem, which may include a trackpad, a touchpad, atrigger, a joystick, a trackball, a rocker or virtual switch, a mouse, akeyboard, a multi-degree-of-freedom controller, or another physicalinput device. A user may use the totem, alone or in combination withposes, to interact with the wearable system or other users.

Examples of haptic devices and totems usable with the wearable devices,HMD, and display systems of the present disclosure are described in U.S.Patent Publication No. 2015/0016777, which is incorporated by referenceherein in its entirety.

E. EXAMPLE OF AN EYE IMAGE

FIG. 5 illustrates an image of an eye 500 with eyelids 504, sclera 508(the “white” of the eye), iris 512, and pupil 516. Curve 516 a shows thepupillary boundary between the pupil 516 and the iris 512, and curve 512a shows the limbic boundary between the iris 512 and the sclera 508. Theeyelids 504 include an upper eyelid 504 a and a lower eyelid 504 b. Theeye 500 is illustrated in a natural resting pose (e.g., in which theuser's face and gaze are both oriented as they would be toward a distantobject directly ahead of the user). The natural resting pose of the eye500 can be indicated by a natural resting direction 520, which is adirection orthogonal to the surface of the eye 500 when in the naturalresting pose (e.g., directly out of the plane for the eye 500 shown inFIG. 5) and in this example, centered within the pupil 516.

As the eye 500 moves to look toward different objects, the eye pose willchange relative to the natural resting direction 520. The current eyepose can be determined with reference to an eye pose direction 524,which is a direction orthogonal to the surface of the eye (and centeredin within the pupil 516) but oriented toward the object at which the eyeis currently directed. With reference to an example coordinate systemshown in FIG. 5A, the pose of the eye 500 can be expressed as twoangular parameters indicating an azimuthal deflection and a zenithaldeflection of the eye pose direction 524 of the eye, both relative tothe natural resting direction 520 of the eye. For purposes ofillustration, these angular parameters can be represented as θ(azimuthal deflection, determined from a fiducial azimuth) and ϕ(zenithal deflection, sometimes also referred to as a polar deflection).In some implementations, angular roll of the eye around the eye posedirection 524 can be included in the determination of eye pose, andangular roll can be included in the following analysis. In otherimplementations, other techniques for determining the eye pose can beused, for example, a pitch, yaw, and optionally roll system.

An eye image can be obtained from a video using any appropriate process,for example, using a video processing algorithm that can extract animage from one or more sequential frames. The pose of the eye can bedetermined from the eye image using a variety of eye-trackingtechniques. For example, an eye pose can be determined by consideringthe lensing effects of the cornea on light sources that are provided.Any suitable eye tracking technique can be used for determining eyepose.

F. EXAMPLE OF AN EYE TRACKING SYSTEM

FIG. 6 illustrates a schematic diagram of a wearable system 600 thatincludes an eye tracking system. The wearable system 600 may, in atleast some embodiments, include components located in a head-mountedunit 602 and components located in a non-head-mounted unit 604. Non-headmounted unit 604 may be, as examples, a belt-mounted component, ahand-held component, a component in a backpack, a remote component, etc.Incorporating some of the components of the wearable system 600 innon-head-mounted unit 604 may help to reduce the size, weight,complexity, and cost of the head-mounted unit 602. In someimplementations, some or all of the functionality described as beingperformed by one or more components of head-mounted unit 602 and/ornon-head mounted 604 may be provided by way of one or more componentsincluded elsewhere in the wearable system 600. For example, some or allof the functionality described below in association with a CPU 612 ofhead-mounted unit 602 may be provided by way of a CPU 616 of non-headmounted unit 604, and vice versa. In some examples, some or all of suchfunctionality may be provided by way of peripheral devices of wearablesystem 600. Furthermore, in some implementations, some or all of suchfunctionality may be provided by way of one or more cloud computingdevices or other remotely-located computing devices in a manner similarto that which has been described above with reference to FIG. 2.

As shown in FIG. 6, wearable system 600 can include an eye trackingsystem including a camera 324 that captures images of a user's eye 610.If desired, the eye tracking system may also include light sources 326 aand 326 b (such as light emitting diodes “LED”s). The light sources 326a and 326 b may generate glints (e.g., reflections off of the user'seyes that appear in images of the eye captured by camera 324). Thepositions of the light sources 326 a and 326 b relative to the camera324 may be known and, as a consequence, the positions of the glintswithin images captured by camera 324 may be used in tracking the user'seyes (as will be discussed in more detail below in connection with FIG.7). In at least one embodiment, there may be one light source 326 andone camera 324 associated with a single one of the user's eyes 610. Inanother embodiment, there may be one light source 326 and one camera 324associated with each of a user's eyes. 610. In yet other embodiments,there may be one or more cameras 324 and one or more light sources 326associated with one or each of a user's eyes 610. As a specific example,there may be two light sources 326 a and 326 b and one or more cameras324 associated with each of a user's eyes 610. As another example, theremay be three or more light sources such as light sources 326 a and 326 band one or more cameras 324 associated with each of a user's eyes 610.In some implementations described herein, two or more cameras may beemployed for imaging a given eye.

Eye tracking module 614 may receive images from eye tracking camera(s)324 and may analyze the images to extract various pieces of information.As examples, the eye tracking module 614 may detect the user's eyeposes, a three-dimensional position of the user's eye relative to theeye tracking camera 324 (and to the head-mounted unit 602), thedirection one or both of the user's eyes 610 are focused on, the user'svergence depth (e.g., the depth from the user at which the user isfocusing on), the positions of the user's pupils, the positions of theuser's cornea and/or cornea sphere, the center of rotation of one oreach of the user's eyes, and the center of perspective of one or each ofthe user's eyes or any combination thereof. The eye tracking module 614may extract such information using techniques described below inconnection with FIGS. 7-11 and/or 12-21. As shown in FIG. 6, eyetracking module 614 may be a software module implemented using a CPU 612in a head-mounted unit 602.

Although one camera 324 is shown in FIG. 6 imaging an eye, in someimplementations such as discussed herein a plurality of cameras mayimage an eye and be used for measurements such as corneal center and/orcenter of rotation measurements or otherwise used for eye tracking orother purposes.

Data from eye tracking module 614 may be provided to other components inthe wearable system. As an example, such data may be transmitted tocomponents in a non-head-mounted unit 604 such as CPU 616 includingsoftware modules for a light-field render controller 618 and aregistration observer 620.

Render controller 618 may use information from eye tracking module 614to adjust images displayed to the user by render engine 622 (e.g., arender engine that may be a software module in GPU 620 and that mayprovide images to display 220). As an example, the render controller 618may adjust images displayed to the user based on the user's center ofrotation or center of perspective. In particular, the render controller618 may use information on the user's center of perspective to simulatea render camera (e.g., to simulate collecting images from the user'sperspective) and may adjust images displayed to the user based on thesimulated render camera.

A “render camera,” which is sometimes also referred to as a “pinholeperspective camera” (or simply “perspective camera”) or “virtual pinholecamera” (or simply “virtual camera”), is a simulated camera for use inrendering virtual image content possibly from a database of objects in avirtual world. The objects may have locations and orientations relativeto the user or wearer and possibly relative to real objects in theenvironment surrounding the user or wearer. In other words, the rendercamera may represent a perspective within render space from which theuser or wearer is to view 3D virtual contents of the render space (e.g.,virtual objects). The render camera may be managed by a render engine torender virtual images based on the database of virtual objects to bepresented to said eye. The virtual images may be rendered as if takenfrom the perspective the user or wearer. For example, the virtual imagesmay be rendered as if captured by a pinhole camera (corresponding to the“render camera”) having a specific set of intrinsic parameters (e.g.,focal length, camera pixel size, principal point coordinates,skew/distortion parameters, etc.), and a specific set of extrinsicparameters (e.g., translational components and rotational componentsrelative to the virtual world). The virtual images are taken from theperspective of such a camera having a position and orientation of therender camera (e.g., extrinsic parameters of the render camera). Itfollows that the system may define and/or adjust intrinsic and extrinsicrender camera parameters. For example, the system may define aparticular set of extrinsic render camera parameters such that virtualimages may be rendered as if captured from the perspective of a camerahaving a specific location with respect to the user's or wearer's eye soas to provide images that appear to be from the perspective of the useror wearer. The system may later dynamically adjust extrinsic rendercamera parameters on-the-fly so as to maintain registration with saidspecific location. Similarly, intrinsic render camera parameters may bedefined and dynamically adjusted over time. In some implementations, theimages are rendered as if captured from the perspective of a camerahaving an aperture (e.g., pinhole) at a specific location with respectto the user's or wearer's eye (such as the center of perspective orcenter of rotation, or elsewhere).

In some embodiments, the system may create or dynamically repositionand/or reorient one render camera for the user's left eye, and anotherrender camera for the user's right eye, as the user's eyes arephysically separated from one another and thus consistently positionedat different locations. It follows that, in at least someimplementations, virtual content rendered from the perspective of arender camera associated with the viewer's left eye may be presented tothe user through an eyepiece on the left side of a head-mounted display(e.g., head-mounted unit 602), and that virtual content rendered fromthe perspective of a render camera associated with the user's right eyemay be presented to the user through an eyepiece on the right side ofsuch a head-mounted display. Further details discussing the creation,adjustment, and use of render cameras in rendering processes areprovided in U.S. patent application Ser. No. 15/274,823, entitled“METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN3D RECONSTRUCTION,” which is expressly incorporated herein by referencein its entirety for all purposes.

In some examples, one or more modules (or components) of the system 600(e.g., light-field render controller 618, render engine 620, etc.) maydetermine the position and orientation of the render camera withinrender space based on the position and orientation of the user's headand eyes (e.g., as determined based on head pose and eye tracking data,respectively). That is, the system 600 may effectively map the positionand orientation of the user's head and eyes to particular locations andangular positions within a 3D virtual environment, place and orientrender cameras at the particular locations and angular positions withinthe 3D virtual environment, and render virtual content for the user asit would be captured by the render camera. Further details discussingreal world to virtual world mapping processes are provided in U.S.patent application Ser. No. 15/296,869, entitled “SELECTING VIRTUALOBJECTS IN A THREE-DIMENSIONAL SPACE,” which is expressly incorporatedherein by reference in its entirety for all purposes. As an example, therender controller 618 may adjust the depths at which images aredisplayed by selecting which depth plane (or depth planes) are utilizedat any given time to display the images. In some implementations, such adepth plane switch may be carried out through an adjustment of one ormore intrinsic render camera parameters. For example, the light-fieldrender controller 618 may adjust the focal lengths of render cameraswhen executing a depth plane switch or adjustment. As described infurther detail below, depth planes may be switched based on the user'sdetermined vergence or fixation depth.

Registration observer 620 may use information from eye tracking module614 to identify whether the head-mounted unit 602 is properly positionedon a user's head. As an example, the eye tracking module 614 may provideeye location information, such as the positions of the centers ofrotation of the user's eyes, indicative of the three-dimensionalposition of the user's eyes relative to camera 324 and head-mounted unit602 and the eye tracking module 614 may use the location information todetermine if display 220 is properly aligned in the user's field ofview, or if the head-mounted unit 602 (or headset) has slipped or isotherwise misaligned with the user's eyes. As examples, the registrationobserver 620 may be able to determine if the head-mounted unit 602 hasslipped down the user's nose bridge, thus moving display 220 away anddown from the user's eyes (which may be undesirable), if thehead-mounted unit 602 has been moved up the user's nose bridge, thusmoving display 220 closer and up from the user's eyes, if thehead-mounted unit 602 has been shifted left or right relative the user'snose bridge, if the head-mounted unit 602 has been lifted above theuser's nose bridge, or if the head-mounted unit 602 has been moved inthese or other ways away from a desired position or range of positions.In general, registration observer 620 may be able to determine ifhead-mounted unit 602, in general, and displays 220, in particular, areproperly positioned in front of the user's eyes. In other words, theregistration observer 620 may determine if a left display in displaysystem 220 is appropriately aligned with the user's left eye and a rightdisplay in display system 220 is appropriately aligned with the user'sright eye. The registration observer 620 may determine if thehead-mounted unit 602 is properly positioned by determining if thehead-mounted unit 602 is positioned and oriented within a desired rangeof positions and/or orientations relative to the user's eyes.

In at least some embodiments, registration observer 620 may generateuser feedback in the form of alerts, messages, or other content. Suchfeedback may be provided to the user to inform the user of anymisalignment of the head-mounted unit 602, along with optional feedbackon how to correct the misalignment (such as a suggestion to adjust thehead-mounted unit 602 in a particular manner).

Example registration observation and feedback techniques, which may beutilized by registration observer 620, are described in U.S. patentapplication Ser. No. 15/717,747, filed Sep. 27, 2017 (Attorney DocketNo. MLEAP.052A2) and U.S. Provisional Patent Application No. 62/644,321,filed Mar. 16, 2018 (Attorney Docket No. MLEAP.195PR), both of which areincorporated by reference herein in their entirety.

G. EXAMPLE OF AN EYE TRACKING MODULE

A detailed block diagram of an example eye tracking module 614 is shownin FIG. 7A. As shown in FIG. 7A, eye tracking module 614 may include avariety of different submodules, may provide a variety of differentoutputs, and may utilize a variety of available data in tracking theuser's eyes. As examples, eye tracking module 614 may utilize availabledata including eye tracking extrinsics and intrinsics, such as thegeometric arrangements of the eye tracking camera 324 relative to thelight sources 326 and the head-mounted-unit 602; assumed eye dimensions704 such as a typical distance of approximately 4.7 mm between a user'scenter of cornea curvature and the average center of rotation of theuser's eye or typical distances between a user's center of rotation andcenter of perspective; and per-user calibration data 706 such as aparticular user's interpupillary distance. Additional examples ofextrinsics, intrinsics, and other information that may be employed bythe eye tracking module 614 are described in U.S. patent applicationSer. No. 15/497,726, filed Apr. 26, 2017 (Attorney Docket No.MLEAP.023A7), which is incorporated by reference herein in its entirety.

Image preprocessing module 710 may receive images from an eye camerasuch as eye camera 324 and may perform one or more preprocessing (e.g.,conditioning) operations on the received images. As examples, imagepreprocessing module 710 may apply a Gaussian blur to the images, maydown sample the images to a lower resolution, may applying an unsharpmask, may apply an edge sharpening algorithm, or may apply othersuitable filters that assist with the later detection, localization, andlabelling of glints, a pupil, or other features in the images from eyecamera 324. The image preprocessing module 710 may apply a low-passfilter or a morphological filter such as an open filter, which canremove high-frequency noise such as from the pupillary boundary 516 a(see FIG. 5), thereby removing noise that can hinder pupil and glintdetermination. The image preprocessing module 710 may outputpreprocessed images to the pupil identification module 712 and to theglint detection and labeling module 714.

Pupil identification module 712 may receive preprocessed images from theimage preprocessing module 710 and may identify regions of those imagesthat include the user's pupil. The pupil identification module 712 may,in some embodiments, determine the coordinates of the position, orcoordinates, of the center, or centroid, of the user's pupil in the eyetracking images from camera 324. In at least some embodiments, pupilidentification module 712 may identify contours in eye tracking images(e.g., contours of pupil iris boundary), identify contour moments (e.g.,centers of mass), apply a starburst pupil detection and/or a canny edgedetection algorithm, reject outliers based on intensity values, identifysub-pixel boundary points, correct for eye-camera distortion (e.g.,distortion in images captured by eye camera 324), apply a random sampleconsensus (RANSAC) iterative algorithm to fit an ellipse to boundariesin the eye tracking images, apply a tracking filter to the images, andidentify sub-pixel image coordinates of the user's pupil centroid. Thepupil identification module 712 may output pupil identification data(which may indicate which regions of the preprocessing images module 712identified as showing the user's pupil) to glint detection and labelingmodule 714. The pupil identification module 712 may provide the 2Dcoordinates of the user's pupil (e.g., the 2D coordinates of thecentroid of the user's pupil) within each eye tracking image to glintdetection module 714. In at least some embodiments, pupil identificationmodule 712 may also provide pupil identification data of the same sortto coordinate system normalization module 718.

Pupil detection techniques, which may be utilized by pupilidentification module 712, are described in U.S. Patent Publication No.2017/0053165, published Feb. 23, 2017 and in U.S. Patent Publication No.2017/0053166, published Feb. 23, 2017, each of which is incorporated byreference herein in its entirety.

Glint detection and labeling module 714 may receive preprocessed imagesfrom module 710 and pupil identification data from module 712. Glintdetection module 714 may use this data to detect and/or identify glints(e.g., reflections off of the user's eye of the light from light sources326) within regions of the preprocessed images that show the user'spupil. As an example, the glint detection module 714 may search forbright regions within the eye tracking image, sometimes referred toherein as “blobs” or local intensity maxima, that are in the vicinity ofthe user's pupil. In at least some embodiments, the glint detectionmodule 714 may rescale (e.g., enlarge) the pupil ellipse to encompassadditional glints. The glint detection module 714 may filter glints bysize and/or by intensity. The glint detection module 714 may alsodetermine the 2D positions of each of the glints within the eye trackingimage. In at least some examples, the glint detection module 714 maydetermine the 2D positions of the glints relative to the user's pupil,which may also be referred to as the pupil-glint vectors. Glintdetection and labeling module 714 may label the glints and output thepreprocessing images with labeled glints to the 3D cornea centerestimation module 716. Glint detection and labeling module 714 may alsopass along data such as preprocessed images from module 710 and pupilidentification data from module 712. In some implementations, the glintdetection and labeling module 714 may determine which light source(e.g., from among a plurality of light sources of the system includinginfrared light sources 326 a and 326 b) produced each identified glint.In these examples, the glint detection and labeling module 714 may labelthe glints with information identifying the associated light source andoutput the preprocessing images with labeled glints to the 3D corneacenter estimation module 716.

Pupil and glint detection, as performed by modules such as modules 712and 714, can use any suitable techniques. As examples, edge detectioncan be applied to the eye image to identify glints and pupils. Edgedetection can be applied by various edge detectors, edge detectionalgorithms, or filters. For example, a Canny Edge detector can beapplied to the image to detect edges such as in lines of the image.Edges may include points located along a line that correspond to thelocal maximum derivative. For example, the pupillary boundary 516 a (seeFIG. 5) can be located using a Canny edge detector. With the location ofthe pupil determined, various image processing techniques can be used todetect the “pose” of the pupil 116. Determining an eye pose of an eyeimage can also be referred to as detecting an eye pose of the eye image.The pose can also be referred to as the gaze, pointing direction, or theorientation of the eye. For example, the pupil may be looking leftwardstowards an object, and the pose of the pupil could be classified as aleftwards pose. Other methods can be used to detect the location of thepupil or glints. For example, a concentric ring can be located in an eyeimage using a Canny Edge detector. As another example, anintegro-differential operator can be used to find the pupillary orlimbus boundaries of the iris. For example, the Daugmanintegro-differential operator, the Hough transform, or other irissegmentation techniques can be used to return a curve that estimates theboundary of the pupil or the iris.

3D cornea center estimation module 716 may receive preprocessed imagesincluding detected glint data and pupil identification data from modules710, 712, 714. 3D cornea center estimation module 716 may use these datato estimate the 3D position of the user's cornea. In some embodiments,the 3D cornea center estimation module 716 may estimate the 3D positionof an eye's center of cornea curvature or a user's corneal sphere, e.g.,the center of an imaginary sphere having a surface portion generallycoextensive with the user's cornea. The 3D cornea center estimationmodule 716 may provide data indicating the estimated 3D coordinates ofthe corneal sphere and/or user's cornea to the coordinate systemnormalization module 718, the optical axis determination module 722,and/or the light-field render controller 618. Further details of theoperation of the 3D cornea center estimation module 716 are providedherein in connection with FIGS. 11-16C. Example techniques forestimating the positions of eye features such as a cornea or cornealsphere, which may be utilized by 3D cornea center estimation module 716and other modules in the wearable systems of the present disclosure arediscussed in U.S. patent application Ser. No. 15/497,726, filed Apr. 26,2017 (Attorney Docket No. MLEAP.023A7), which is incorporated byreference herein in its entirety.

Coordinate system normalization module 718 may optionally (as indicatedby its dashed outline) be included in eye tracking module 614.Coordinate system normalization module 718 may receive data indicatingthe estimated 3D coordinates of the center of the user's cornea (and/orthe center of the user's corneal sphere) from the 3D cornea centerestimation module 716 and may also receive data from other modules.Coordinate system normalization module 718 may normalize the eye cameracoordinate system, which may help to compensate for slippages of thewearable device (e.g., slippages of the head-mounted component from itsnormal resting position on the user's head, which may be identified byregistration observer 620). Coordinate system normalization module 718may rotate the coordinate system to align the z-axis (e.g., the vergencedepth axis) of the coordinate system with the cornea center (e.g., asindicated by the 3D cornea center estimation module 716) and maytranslate the camera center (e.g., the origin of the coordinate system)to a predetermined distance away from the cornea center such as 30 mm(e.g., module 718 may enlarge or shrink the eye tracking image dependingon whether the eye camera 324 was determined to be nearer or furtherthan the predetermined distance). With this normalization process, theeye tracking module 614 may be able to establish a consistentorientation and distance in the eye tracking data, relativelyindependent of variations of headset positioning on the user's head.Coordinate system normalization module 718 may provide 3D coordinates ofthe center of the cornea (and/or corneal sphere), pupil identificationdata, and preprocessed eye tracking images to the 3D pupil centerlocator module 720.

3D pupil center locator module 720 may receive data, in the normalizedor the unnormalized coordinate system, including the 3D coordinates ofthe center of the user's cornea (and/or corneal sphere), pupil locationdata, and preprocessed eye tracking images. 3D pupil center locatormodule 720 may analyze such data to determine the 3D coordinates of thecenter of the user's pupil in the normalized or unnormalized eye cameracoordinate system. The 3D pupil center locator module 720 may determinethe location of the user's pupil in three-dimensions based on the 2Dposition of the pupil centroid (as determined by module 712), the 3Dposition of the cornea center (as determined by module 716), assumed eyedimensions 704 such as the size of the a typical user's corneal sphereand the typical distance from the cornea center to the pupil center, andoptical properties of eyes such as the index of refraction of the cornea(relative to the index of refraction of air) or any combination ofthese. Techniques for estimating the positions of eye features such as apupil, which may be utilized by 3D pupil center locator module 720 andother modules in the wearable systems of the present disclosure arediscussed in U.S. patent application Ser. No. 15/497,726, filed Apr. 26,2017 (Attorney Docket No. MLEAP.023A7), which is incorporated byreference herein in its entirety.

Optical axis determination module 722 may receive data from modules 716and 720 indicating the 3D coordinates of the center of the user's corneaand the user's pupil. Based on such data, the optical axis determinationmodule 722 may identify a vector from the position of the cornea center(e.g., from the center of the corneal sphere) to the center of theuser's pupil, which may define the optical axis of the user's eye.Optical axis determination module 722 may provide outputs specifying theuser's optical axis to modules 724, 728, 730, and 732, as examples.

Center of rotation (CoR) estimation module 724 may receive data frommodule 722 including parameters of the optical axis of the user's eye(e.g., data indicating the direction of the optical axis in a coordinatesystem with a known relation to the head-mounted unit 602). For example,CoR estimation module 724 may estimate the center of rotation of auser's eye. The center of rotation may indicate a point around which theuser's eye rotates when the user eye rotates left, right, up, and/ordown. While eyes may not rotate perfectly around a singular point,assuming a singular point may be sufficient. In at least someembodiments, CoR estimation module 724 may estimate an eye's center ofrotation by moving from the center of the pupil (identified by module720) or the center of curvature of the cornea (as identified by module716) toward the retina along the optical axis (identified by module 722)a particular distance. This particular distance may be an assumed eyedimension 704. As one example, the particular distance between thecenter of curvature of the cornea and the CoR may be approximately 4.7mm. This distance may be varied for a particular user based on anyrelevant data including the user's age, sex, vision prescription, otherrelevant characteristics, etc. As discussed above, In someimplementations, the center of curvature of the cornea or the center ofthe cornea refers to the center of curvature of a portion of the corneaor the center of curvature of a spherical surface that coincides with aportion of the surface of the cornea. For example, in someimplementations, the center of curvature of the cornea or the center ofthe cornea refers to the center of curvature of the cornea apex or thecenter of curvature of a spherical surface that coincides with a portionof the surface of the corneal apex.

In at least some embodiments, the CoR estimation module 724 may refineits estimate of the center of rotation of each of the user's eyes overtime. As an example, as time passes, the user will eventually rotatetheir eyes (to look somewhere else, at something closer, further, orsometime left, right, up, or down) causing a shift in the optical axisof each of their eyes. CoR estimation module 724 may then analyze two(or more) optical axes identified by module 722 and locate the 3D pointof intersection of those optical axes. The CoR estimation module 724 maythen determine the center of rotation lies at that 3D point ofintersection. Such a technique may provide for an estimate of the centerof rotation, with an accuracy that improves over time.

Various techniques may be employed to increase the accuracy of the CoRestimation module 724 and the determined CoR positions of the left andright eyes. As an example, the CoR estimation module 724 may estimatethe CoR by finding the average point of intersection of optical axesdetermined for various different eye poses over time. As additionalexamples, module 724 may filter or average estimated CoR positions overtime, may calculate a moving average of estimated CoR positions overtime, and/or may apply a Kalman filter and known dynamics of the eyesand eye tracking system to estimate the CoR positions over time. In someimplementations, a least-squares approach may be taken to determine oneor more points of intersection of optical axes. In such implementations,the system may, at a given point in time, identify a location at whichthe sum of the squared distances to a given set of optical axes isreduced or minimized as the point of optical axes intersection. As aspecific example, module 724 may calculate a weighted average ofdetermined points of optical axes intersection and assumed CoR positions(such as 4.7 mm from an eye's center of cornea curvature), such that thedetermined CoR may slowly drift from an assumed CoR position (e.g., 4.7mm behind an eye's center of cornea curvature) to a slightly differentlocation within the user's eye over time as eye tracking data for theuser is obtain and thereby enables per-user refinement of the CoRposition.

Under ideal conditions, the 3D position of the true CoR of a user's eyerelative to the HMD should change a negligible or minimal amount overtime as the user moves their eye (e.g., as the user's eye rotates aroundits center of rotation). In other words, for a given set of eyemovements, the 3D position of the true CoR of the user's eye (relativeto the HMD) should hypothetically vary less over time than any otherpoint along the optical axis of the user's eye. As such, it follows thatthe further away a point along the optical axis is from the true CoR ofthe user's eye, the more variation or variance its 3D position willexhibit over time as the user moves their eye. In some embodiments, theCoR estimation module 724 and/or other submodules of eye tracking module614 may make use of this statistical relationship to improve CoRestimation accuracy. In such embodiments, the CoR estimation module 724and/or other submodules of eye tracking module 614 may refine theirestimates of the CoR 3D position over time by identifying variations ofits CoR estimates having a low variation (e.g., low variance or standarddeviation).

As a first example and in embodiments where the CoR estimation module724 estimates CoR based on intersection of multiple different opticalaxes (each associated with the user looking in a different direction),the CoR estimation module 724 may make use of this statisticalrelationship (that the true CoR should have a low variance) byintroducing common offsets to the direction of each of the optical axes(e.g., shifting each axis by some uniform amount) and determining if theoffset optical axes intersect with each other in an intersection pointhaving a low variation, e.g., low variance or standard deviation. Thismay correct for minor systemic errors in calculating the directions ofthe optical axes and help to refine the estimated position of the CoR tobe closer to the true CoR.

As a second example and in embodiments where the CoR estimation module724 estimates CoR by moving along an optical axis (or other axis) by aparticular distance (e.g., such as the distance between the center ofcurvature of the cornea and the CoR), the system may vary, optimize,tune, or otherwise adjust the particular distance between the center ofcurvature of the cornea and the CoR over time (for example, for a largegroup of images of the eye captured at different times) in a manner soas to reduce or minimize the variation, for example, variance and/orstandard deviation of the estimated CoR position. For example, if theCoR estimation module 724 initially uses a particular distance value of4.7 mm (from the center of curvature of the cornea and along the opticalaxis) to obtain CoR position estimates, but the true CoR of a givenuser's eye may be positioned 4.9 mm behind the eye's center of corneacurvature (along the optical axis), then an initial set of CoR positionestimates obtained by the CoR estimation module 724 may exhibit arelatively high amount of variation, e.g., variance or standarddeviation. In response to detecting such a relatively high amount ofvariation (e.g., variance or standard deviation), the CoR estimationmodule 724 may look for and identify one or more points along theoptical axis having a lower amount of variation (e.g., variance orstandard deviation), may identify the 4.9 mm distance as having thelowest variation (e.g., variance or standard deviation), and may thusadjust the particular distance value utilized to 4.9 mm.

The CoR estimation module 724 may look for alternative CoR estimationshaving lower variation (e.g., variance and/or standard deviation) inresponse to detecting that a current CoR estimate has a relatively highamount of variation (e.g., variance or standard deviation) or may lookfor alternative CoR estimations having lower variation (e.g. variance orstandard deviation) as a matter of course after obtaining initial CoRestimates. In some examples, such an optimization/adjustment can happengradually over time, while in other examples, such anoptimization/adjustment can be made during an initial user calibrationsession. In examples where such a procedure is conducted during acalibration procedure, the CoR estimation module 724 may not initiallysubscribe/adhere to any assumed particular distance, but may rathercollect a set of eye tracking data over time, perform statisticalanalysis on the set of eye tracking data, and determine the particulardistance value yielding CoR position estimates with the least possibleamount (e.g., global minima) of variation (e.g. variance or standarddeviation) based on the statistical analysis.

Interpupillary distance (IPD) estimation module 726 may receive datafrom CoR estimation module 724 indicating the estimated 3D positions ofthe centers of rotation of the user's left and right eyes. IPDestimation module 726 may then estimate a user's IPD by measuring the 3Ddistance between the centers of rotation of the user's left and righteyes. In general, the distance between the estimated CoR of the user'sleft eye and the estimated CoR of the user's right eye may be roughlyequal to the distance between the centers of a user's pupils, when theuser is looking at optical infinity (e.g., the optical axes of theuser's eyes are substantially parallel to one another), which is thetypical definition of interpupillary distance (IPD). A user's IPD may beused by various components and modules in the wearable system. Asexample, a user's IPD may be provided to registration observer 620 andused in assessing how well the wearable device is aligned with theuser's eyes (e.g., whether the left and right display lenses areproperly spaced in accordance with the user's IPD). As another example,a user's IPD may be provided to vergence depth estimation module 728 andbe used in determining a user's vergence depth. Module 726 may employvarious techniques, such as those discussed in connection with CoRestimation module 724, to increase the accuracy of the estimated IPD. Asexamples, IPD estimation module 724 may apply filtering, averaging overtime, weighted averaging including assumed IPD distances, Kalmanfilters, etc. as part of estimating a user's IPD in an accurate manner.

Vergence depth estimation module 728 may receive data from variousmodules and submodules in the eye tracking module 614 (as shown inconnection with FIG. 7A). In particular, vergence depth estimationmodule 728 may employ data indicating estimated 3D positions of pupilcenters (e.g., as provided by module 720 described above), one or moredetermined parameters of optical axes (e.g., as provided by module 722described above), estimated 3D positions of centers of rotation (e.g.,as provided by module 724 described above), estimated IPD (e.g.,Euclidean distance(s) between estimated 3D positions of centers ofrotations) (e.g., as provided by module 726 described above), and/or oneor more determined parameters of optical and/or visual axes (e.g., asprovided by module 722 and/or module 730 described below). Vergencedepth estimation module 728 may detect or otherwise obtain a measure ofa user's vergence depth, which may be the distance from the user atwhich the user's eyes are focused. As examples, when the user is lookingat an object three feet in front of them, the user's left and right eyeshave a vergence depth of three feet; and, while when the user is lookingat a distant landscape (e.g., the optical axes of the user's eyes aresubstantially parallel to one another such that the distance between thecenters of the user's pupils may be roughly equal to the distancebetween the centers of rotation of the user's left and right eyes), theuser's left and right eyes have a vergence depth of infinity. In someimplementations, the vergence depth estimation module 728 may utilizedata indicating the estimated centers of the user's pupils (e.g., asprovided by module 720) to determine the 3D distance between theestimated centers of the user's pupils. The vergence depth estimationmodule 728 may obtain a measure of vergence depth by comparing such adetermined 3D distance between pupil centers to estimated IPD (e.g.,Euclidean distance(s) between estimated 3D positions of centers ofrotations) (e.g., as indicated by module 726 described above). Inaddition to the 3D distance between pupil centers and estimated IPD, thevergence depth estimation module 728 may utilize known, assumed,estimated, and/or determined geometries to calculate vergence depth. Asan example, module 728 may combine 3D distance between pupil centers,estimated IPD, and 3D CoR positions in a trigonometric calculation toestimate (e.g., determine) a user's vergence depth. Indeed, anevaluation of such a determined 3D distance between pupil centersagainst estimated IPD may serve to indicate a measure of the user'scurrent vergence depth relative to optical infinity. In some examples,the vergence depth estimation module 728 may simply receive or accessdata indicating an estimated 3D distance between the estimated centersof the user's pupils for purposes of obtaining such a measure ofvergence depth. In some embodiments, the vergence depth estimationmodule 728 may estimate vergence depth by comparing a user's left andright optical axis. In particular, vergence depth estimation module 728may estimate vergence depth by locating the distance from a user atwhich the user's left and right optical axes intersect (or whereprojections of the user's left and right optical axes on a plane such asa horizontal plane intersect). Module 728 may utilize a user's IPD inthis calculation, by setting the zero depth to be the depth at which theuser's left and right optical axes are separated by the user's IPD. Inat least some embodiments, vergence depth estimation module 728 maydetermine vergence depth by triangulating eye tracking data togetherwith known or derived spatial relationships.

In some embodiments, vergence depth estimation module 728 may estimate auser's vergence depth based on the intersection of the user's visualaxes (instead of their optical axes), which may provide a more accurateindication of the distance at which the user is focused on. In at leastsome embodiments, eye tracking module 614 may include optical to visualaxis mapping module 730. As discussed in further detail in connectionwith FIG. 10, a user's optical and visual axes are generally notaligned. A visual axis is the axis along which a person is looking,while an optical axis is defined by the center of that person's lens andpupil, and may go through the center of the person's retina. Inparticular, a user's visual axis is generally defined by the location ofthe user's fovea, which may be offset from the center of a user'sretina, thereby resulting in different optical and visual axis. In atleast some of these embodiments, eye tracking module 614 may includeoptical to visual axis mapping module 730. Optical to visual axismapping module 730 may correct for the differences between a user'soptical and visual axis and provide information on the user's visualaxis to other components in the wearable system, such as vergence depthestimation module 728 and light-field render controller 618. In someexamples, module 730 may use assumed eye dimensions 704 including atypical offset of approximately 5.2° inwards (nasally, towards a user'snose) between an optical axis and a visual axis. In other words, module730 may shift a user's left optical axis (nasally) rightwards by 5.2°towards the nose and a user's right optical axis (nasally) leftwards by5.2° towards the nose in order to estimate the directions of the user'sleft and right optical axes. In other examples, module 730 may utilizeper-user calibration data 706 in mapping optical axes (e.g., asindicated by module 722 described above) to visual axes. As additionalexamples, module 730 may shift a user's optical axes nasally by between4.0° and 6.5°, by between 4.5° and 6.0°, by between 5.0° and 5.4°, etc.,or any ranges formed by any of these values. In some arrangements, themodule 730 may apply a shift based at least in part upon characteristicsof a particular user such as their age, sex, vision prescription, orother relevant characteristics and/or may apply a shift based at leastin part upon a calibration process for a particular user (e.g., todetermine a particular user's optical-visual axis offset). In at leastsome embodiments, module 730 may also shift the origins of the left andright optical axes to correspond with the user's CoP (as determined bymodule 732) instead of the user's CoR.

Optional center of perspective (CoP) estimation module 732, whenprovided, may estimate the location of the user's left and right centersof perspective (CoP). A CoP may be a useful location for the wearablesystem and, in at least some embodiments, is a position just in front ofa pupil. In at least some embodiments, CoP estimation module 732 mayestimate the locations of a user's left and right centers of perspectivebased on the 3D location of a user's pupil center, the 3D location of auser's center of cornea curvature, or such suitable data or anycombination thereof. As an example, a user's CoP may be approximately5.01 mm in front of the center of cornea curvature (e.g., 5.01 mm fromthe corneal sphere center in a direction that is towards the eye'scornea and that is along the optical axis) and may be approximately 2.97mm behind the outer surface of a user's cornea, along the optical orvisual axis. A user's center of perspective may be just in front of thecenter of their pupil. As examples, a user's CoP may be less thanapproximately 2.0 mm from the user's pupil, less than approximately 1.0mm from the user's pupil, or less than approximately 0.5 mm from theuser's pupil or any ranges between any of these values. As anotherexample, the center of perspective may correspond to a location withinthe anterior chamber of the eye. As other examples, the CoP may bebetween 1.0 mm and 2.0 mm, about 1.0 mm, between 0.25 mm and 1.0 mm,between 0.5 mm and 1.0 mm, or between 0.25 mm and 0.5 mm from the user'spupil.

The center of perspective described herein (as a potentially desirableposition for a pinhole of a render camera and an anatomical position ina user's eye) may be a position that serves to reduce and/or eliminateundesired parallax shifts. In particular, the optical system of a user'seye is very roughly equivalent to theoretical system formed by a pinholein front of a lens, projecting onto a screen, with the pinhole, lens,and screen roughly corresponding to a user's pupil/iris, lens, andretina, respectively. Moreover, it may be desirable for there to belittle or no parallax shift when two point light sources (or objects) atdifferent distances from the user's eye are rigidly rotated about theopening of the pinhole (e.g., rotated along radii of curvature equal totheir respective distance from the opening of the pinhole). Thus, itwould seem that the CoP should be located at the center of the pupil ofan eye (and such a CoP may be used in some embodiments). However, thehuman eye includes, in addition to the lens and pinhole of the pupil, acornea that imparts additional optical power to light propagating towardthe retina). Thus, the anatomical equivalent of the pinhole in thetheoretical system described in this paragraph may be a region of theuser's eye positioned between the outer surface of the cornea of theuser's eye and the center of the pupil or iris of the user's eye. Forinstance, the anatomical equivalent of the pinhole may correspond to aregion within the anterior chamber of a user's eye. For various reasonsdiscussed herein, it may be desired to set the CoP to such a positionwithin the anterior chamber of the user's eye.

As discussed above, eye tracking module 614 may provide data, such asestimated 3D positions of left and right eye centers of rotation (CoR),vergence depth, left and right eye optical axis, 3D positions of auser's eye, 3D positions of a user's left and right centers of corneacurvature, 3D positions of a user's left and right pupil centers, 3Dpositions of a user's left and right center of perspective, a user'sIPD, etc., to other components, such as light-field render controller618 and registration observer 620, in the wearable system. Eye trackingmodule 614 may also include other submodules that detect and generatedata associated with other aspects of a user's eye. As examples, eyetracking module 614 may include a blink detection module that provides aflag or other alert whenever a user blinks and a saccade detectionmodule that provides a flag or other alert whenever a user's eyesaccades (e.g., quickly shifts focus to another point).

Other methods of eye tracking and determining the center of rotation arepossible. Accordingly, the eye tracking module 614 may be different. Invarious implementations of eye tracking modules described below, forexample, estimates of center of rotation are determined based on aplurality of center of corneal curvature values. In someimplementations, for example, as discussed with reference to FIGS.17A-19D, the eye tracking module 614 may estimate an eye's center ofrotation by determining an convergence or intersection among surfacenormal vectors of a surface fitted to a plurality of center ofcurvatures of the cornea possibly for different eye poses. Nevertheless,one or more features from the eye tracking module 614 described above orelsewhere herein may be included in other implementations of eyetracking modules.

H. EXAMPLE OF A RENDER CONTROLLER

A detailed block diagram of an example light-field render controller 618is shown in FIG. 7B. As shown in FIGS. 6 and 7B, render controller 618may receive eye tracking information from eye tracking module 614 andmay provide outputs to render engine 622, which may generate images tobe displayed for viewing by a user of the wearable system. As examples,render controller 618 may receive a vergence depth, left and right eyecenters of rotation (and/or centers of perspective), and other eye datasuch as blink data, saccade data, etc.

Depth plane selection module 750 may receive vergence depth informationand other eye data and, based on such data, may cause render engine 622to convey content to a user with a particular depth plane (e.g., at aparticular accommodation or focal distance). As discussed in connectionwith FIG. 4, a wearable system may include a plurality of discrete depthplanes formed by a plurality of waveguides, each conveying imageinformation with a varying level of wavefront curvature. In someembodiments, a wearable system may include one or more variable depthplanes, such as an optical element that conveys image information with alevel of wavefront curvature that varies over time. In these and otherembodiments, depth plane selection module 750 may cause render engine622 to convey content to a user at a selected depth (e.g., cause renderengine 622 to direct display 220 to switch depth planes), based in partof the user's vergence depth. In at least some embodiments, depth planeselection module 750 and render engine 622 may render content atdifferent depths and also generate and/or provide depth plane selectiondata to display hardware such as display 220. Display hardware such asdisplay 220 may perform an electrical depth plane switching in responseto depth plane selection data (which may be control signals) generatedby and/or provided by modules such as depth plane selection module 750and render engine 622.

In general, it may be desirable for depth plane selection module 750 toselect a depth plane matching the user's current vergence depth, suchthat the user is provided with accurate accommodation cues. However, itmay also be desirable to switch depth planes in a discreet andunobtrusive manner. As examples, it may be desirable to avoid excessiveswitching between depth planes and/or it may be desire to switch depthplanes at a time when the user is less likely to notice the switch, suchas during a blink or eye saccade.

Hysteresis band crossing detection module 752 may help to avoidexcessive switching between depth planes, particularly when a user'svergence depth fluctuates at the midpoint or transition point betweentwo depth planes. In particular, module 752 may cause depth planeselection module 750 to exhibit hysteresis in its selection of depthplanes. As an example, modules 752 may cause depth plane selectionmodule 750 to switch from a first farther depth plane to a second closerdepth plane only after a user's vergence depth passes a first threshold.Similarly, module 752 may cause depth plane selection module 750 (whichmay in turn direct displays such as display 220) to switch to the firstfarther depth plane only after the user's vergence depth passes a secondthreshold that is farther from the user than the first threshold. In theoverlapping region between the first and second thresholds, module 750may cause depth plane selection module 750 to maintain whichever depthplane is currently select as the selected depth plane, thus avoidingexcessive switching between depth planes.

Ocular event detection module 750 may receive other eye data from theeye tracking module 614 of FIG. 7A and may cause depth plane selectionmodule 750 to delay some depth plane switches until an ocular eventoccurs. As an example, ocular event detection module 750 may cause depthplane selection module 750 to delay a planned depth plane switch until auser blink is detected; may receive data from a blink detectioncomponent in eye tracking module 614 that indicates when the user iscurrently blinking; and, in response, may cause depth plane selectionmodule 750 to execute the planned depth plane switch during the blinkevent (such by causing module 750 to direct display 220 to execute thedepth plane switch during the blink event). In at least someembodiments, the wearable system may be able to shift content onto a newdepth plane during a blink event such that the user is unlikely toperceive the shift. As another example, ocular event detection module750 may delay planned depth plane switches until an eye saccade isdetected. As discussed in connection with eye blinks, such as anarrangement may facilitate the discretely shifting of depth planes.

If desired, depth plane selection module 750 may delay planned depthplane switches only for a limited period of time before executing thedepth plane switch, even in the absence of an ocular event. Similarly,depth plane selection module 750 may execute a depth plane switch whenthe user's vergence depth is substantially outside of acurrently-selected depth plane (e.g., when the user's vergence depth hasexceeded a predetermined threshold beyond the regular threshold for adepth plane switch), even in the absence of an ocular event. Thesearrangements may help ensure that ocular event detection module 754 doesnot indefinitely delay depth plane switches and does not delay depthplane switches when a large accommodation error is present.

Render camera controller 758 may provide information to render engine622 indicating where the user's left and right eyes are. Render engine622 may then generate content by simulating cameras at the positions ofthe user's left and right eyes and generating content based on theperspectives of the simulated cameras. As discussed above, the rendercamera is a simulated camera for use in rendering virtual image contentpossibly from a database of objects in a virtual world. The objects mayhave locations and orientations relative to the user or wearer andpossibly relative to real objects in the environment surrounding theuser or wearer. The render camera may be included in a render engine torender virtual images based on the database of virtual objects to bepresented to said eye. The virtual images may be rendered as if takenfrom the perspective the user or wearer. For example, the virtual imagesmay be rendered as if captured by a camera (corresponding to the “rendercamera”) having an aperture, lens, and detector viewing the objects inthe virtual world. The virtual images are taken from the perspective ofsuch a camera having a position of the “render camera.” For example, thevirtual images may be rendered as if captured from the perspective of acamera having a specific location with respect to the user's or wearer'seye so as to provide images that appear to be from the perspective ofthe user or wearer. In some implementations, the images are rendered asif captured from the perspective of a camera having an aperture at aspecific location with respect to the user's or wearer's eye (such asthe center of perspective or center of rotation as discussed herein, orelsewhere).

Render camera controller 758 may determine the positions of the left andright cameras based on the left and right eye centers of rotation (CoR),determined by CoR estimation module 724, and/or based on the left andright eye centers of perspective (CoP), determined by CoP estimationmodule 732. In some embodiments, render camera controller 758 may switchbetween the CoR and CoP locations based on various factors. As examples,the render camera controller 758 may, in various modes, register therender camera to the CoR locations at all times, register the rendercamera to the CoP locations at all times, toggle or discretely switchbetween registering the render camera to the CoR locations andregistering the render camera to the CoP locations over time based onvarious factors, or dynamically register the render camera to any of arange of different positions along the optical (or visual) axis betweenthe CoR and CoP locations over time based on various factors. The CoRand CoP positions may optionally pass through smoothing filter 756 (inany of the aforementioned modes for render camera positioning) which mayaverage the CoR and CoP locations over time to reduce noise in thesepositions and prevent jitter in the render simulated render cameras.

In at least some embodiments, the render camera may be simulated as apinhole camera with the pinhole disposed at the position of theestimated CoR or CoP identified by eye tracking module 614. As the CoPis offset from the CoR, the location of the render camera and itspinhole both shift as the user's eye rotates, whenever the rendercamera's position is based on a user's CoP. In contrast, whenever therender camera's position is based on a user's CoR, the location of therender camera's pinhole does not move with eye rotations, although therender camera (which is behind the pinhole) may, in some embodiments,move with eye rotation. In other embodiments where the render camera'sposition is based on a user's CoR, the render camera may not move (e.g.,rotate) with a user's eye.

I. EXAMPLE OF DIFFERENCES BETWEEN OPTICAL AND VISUAL AXES

As discussed in connection with optical to visual mapping module 730 ofFIG. 7A, a user's optical and visual axes are generally not aligned, duein part to a user's visual axis being defined by their fovea and thatfoveae are not generally in the center of a person's retina. Thus, whena person desires to concentrate on a particular object, the personaligns their visual axis with that object to ensure that light from theobject falls on their fovea while their optical axis (defined by thecenter of their pupil and center of curvature of their cornea) isactually slightly offset from that object. FIG. 8 is an example of aneye 900 illustrating the eye's optical axis 902, the eye's visual axis904, and the offset between these axes. Additionally, FIG. 8 illustratesthe eye's pupil center 906, the eye's center of cornea curvature 908,and the eye's average center of rotation (CoR) 910. In at least somepopulations, the eye's center of cornea curvature 908 may lieapproximately 4.7 mm in front, as indicated by dimension 912, of theeye's average center of rotation (CoR) 910. Additionally, the eye'scenter of perspective 914 may lie approximately 5.01 mm in front of theeye's center of cornea curvature 908, about 2.97 mm behind the outersurface 916 of the user's cornea, and/or just in front of the user'spupil center 906 (e.g., corresponding to a location within the anteriorchamber of eye 900). As additional examples, dimension 912 may between3.0 mm and 7.0 mm, between 4.0 and 6.0 mm, between 4.5 and 5.0 mm, orbetween 4.6 and 4.8 mm or any ranges between any values and any valuesin any of these ranges. The eye's center of perspective (CoP) 914 may bea useful location for the wearable system as, in at least someembodiments, registering a render camera at the CoP may help to reduceor eliminate parallax artifacts.

FIG. 8 also illustrates a location within a human eye 900 with which thepinhole of a render camera can be aligned. As shown in FIG. 8, thepinhole of a render camera may be registered with a location 914 alongthe optical axis 902 or visual axis 904 of the human eye 900 closer tothe outer surface of the cornea than both (a) the center of the pupil oriris 906 and (b) the center of cornea curvature 908 of the human eye900. For example, as shown in FIG. 8, the pinhole of a render camera maybe registered with a location 914 along the optical axis 902 of thehuman eye 900 that is about 2.97 millimeters rearward from the outersurface of the cornea 916 and about 5.01 millimeters forward from thecenter of cornea curvature 908. The location 914 of the pinhole of therender camera and/or the anatomical region of the human eye 900 to whichthe location 914 corresponds can be seen as representing the center ofperspective of the human eye 900. The optical axis 902 of the human eye900 as shown in FIG. 8 represents the most direct line through thecenter of cornea curvature 908 and the center of the pupil or iris 906.The visual axis 904 of the human eye 900 differs from the optical axis902, as it represents a line extending from the fovea of the human eye900 to the center of the pupil or iris 906.

J. EXAMPLE WEARABLE DEVICE CONFIGURATION

FIGS. 9A-E illustrate an example configuration of components of anexample wearable device 2000 for capturing eye image data for use by aneye tracking module 614. For example, as illustrated in FIG. 9A, awearable device 2000 may be part of a wearable system, such as describedabove with reference to FIGS. 2-4. The wearable device 2000 may includea left eye piece 2010A and a right eyepiece 2010B. The left eyepiece2010A may be able to image a user's left eye and the right eyepiece2010B may be able to image a user's right eye.

As illustrated in FIG. 9B, the left eyepiece 2010A may include one ormore illumination sources 2022. Similarly, the right eyepiece 2010B mayinclude one or more illumination sources 2024. For example, there may befour illumination sources 2022 and four illumination sources 2024. Theillumination sources 2022 may be positioned within a left eyepiece 2010Ato emit light towards a user's left eye 2012A. The illumination sources2022 may be positioned so as not to obstruct the user's view through theleft eyepiece 2010A. For example, the illumination sources 2022 may bepositioned around a rim of a display within the left eyepiece 2010A soas not to obstruct a user's view through the display. Similarly, theillumination sources 2024 may be positioned within a right eyepiece2010B to emit light towards a user's right eye 2012B. The illuminationsources 2024 may be positioned so as not to obstruct the user's viewthrough the right eyepiece 2010B. For example, the illumination sources20204 may be positioned around a rim of a display within the righteyepiece 2010B so as not to obstruct a user's view through the display.The illumination sources 2022, 2024 may emit light in visible ornon-visible light. For example, the illumination sources 2022, 2024 maybe infrared (IR) LEDs. The illumination sources may also be located orconfigured differently.

As illustrated in FIG. 9B, the left eye piece 2010A may include a lefteye imaging system. The left eye imaging system can include one or moreinward-facing cameras (2014, 2016). For example, the left eye imagingsystem can include a left eye tracking camera 2014 for the left eyepiece2010A and a right eye tracking camera 2016 for the left eyepiece 2010A.The left eye tracking camera 2014 and right eye tracking camera 2016 maybe located to the left and right of each other, respectively, possiblyleft and right of center of the left eyepiece, respectively. Similarly,the right eye piece 2010B may include a right eye imaging system. Theright eye imaging system can include one or more inward-facing cameras(2018, 2020). For example, the right eye imaging system can include aleft eye tracking camera 2018 for the right eyepiece 2010B and a righteye tracking camera 2020 for the right eyepiece 2010B. The left eyetracking camera 2018 and right eye tracking camera 2020 may be locatedto the left and right of each other, respectively, possibly left andright of center of the right eyepiece, respectively. The one or morecameras in the left eye tracking system 2010A and the one or morecameras in the right eye tracking system 2010B may be situated withinthe wearable device 2000 so as to unobtrusively capture images of theuser's eye(s). Other configurations are possible.

The fields of view of the imaging system for the left eyepiece 2010A canbe capable of imaging all or a useful portion of the user's left eye2012A in many different eye pose positions (and may not necessarilyimage the right eye or a portion thereof useful for eye tracking).Similarly, the fields of view of the imaging system for the righteyepiece 2010B can be capable of imaging all or a useful portion of theuser's right eye 2012B in many different eye pose positions (and may notnecessarily image the left eye or a portion thereof useful for eyetracking). For example, a user may be able move their eye up to 50degrees from center gaze in any direction during normal movement. Theimaging systems may be situated to collectively image substantially allof the full range of motion (e.g., of 50 degrees) of the user's eyesduring their normal movement.

FIG. 9C illustrates an example field of view 2030 of the right eyetracking camera 2016 of the left eye piece 2010A and an example field ofview 2032 of the right eye tracking camera 2020 of the right eye piece2010B. FIG. 9D illustrates an example field of view 2040 of the righteye tracking camera 2014 of the left eye piece 2010A and an examplefield of view 2042 of the right eye tracking camera 2018 of the righteye piece 2010B. FIG. 9E illustrates how fields of view 2030 and 2040from the left eye tracking camera 2014 and right eye tracking camera2016 of the left eyepiece 2010A respectively can overlap so as to imagesubstantially all of the user's left eye 2012A. Additionally, FIG. 9Eillustrates how fields of view 2040 and 2042 from the left eye trackingcamera 2018 and right eye tracking camera 2020 of the right eyepiece2010B respectively can overlap so as to image substantially all of theuser's right eye 2012B. Variations are possible. For example, the numberand locations of the cameras can be different. Other types of imagingsystems may also be used.

K. EXAMPLE OF LOCATING A CENTER OF ROTATION WITH AN EYE TRACKING SYSTEM

In order to simplify an eye tracking system (or processes within an eyetracking module 614), it may be desirable to reduce the number ofvariables required to determine a Center of Rotation (CoR) of the humaneye. Advantageously, reducing the number of variables used to determinea CoR can also improve eye tracking accuracy. For example, since the CoRmay be used to determine a gaze vector for use in eye tracking,increased error in the CoR may result in less accurate eye tracking.Error in the CoR may result from errors introduced during determinationof variables used for calculating the CoR. For example, a CoRcalculation may involve extracting a pupil disk center and modeling acorneal sphere. Both of those processes may introduce error andcontribute to inaccuracy. Thus, it may be advantageous to extract a CoRusing a limited number of variables.

Described herein are systems and methods for extracting a CoR primarilyor entirely from corneal data. Advantageously, due to similar reasons asthose discussed above, the present system can improve accuracy of an eyetracking system. For example, the present system may require fewassumptions, thus reducing the potential for introduction of error.Additionally or in the alternative to improved accuracy, the presentsystem can improve other aspects of an eye tracking system. For example,the present system may rely on shorter eye exposure to illuminationsources. The shorter eye exposure can reduce risks associated withprolonged eye exposure to an illumination source, reduce illuminationpower consumption, and provide high ambient light rejection. In anotherexample, the present system may not require a large field of view. Thereduced field of view requirement can allow for greater flexibility inhardware design of a wearable system.

In some examples, the Center of Rotation (CoR) of a human eye can beextracted from corneal data. FIG. 10 shows a graphical illustration ofan example CoR extraction system 1000 that may be performed by an eyetracking module 614. For example, a wearable system may generate two ormore glints on the cornea 1020 of a user's eye 1010 using anillumination system comprising a one or more illumination sources. Invarious implementations, the illumination system comprises a pluralityof separate regions where light is output. These regions may correspondto separated light emitters or light sources. A glint detection andlabeling module 714 may extract the glint location(s) on the cornea 1020of the eye 1010. As described below, a 3D cornea center estimationmodule 716 may determine an approximate corneal curvature 1018 based onthe glint locations and calculate an estimated center 1012 of thatapproximated corneal curvature 1018. Different eye poses may providedifferent approximated corneal curvatures 1018 and associated estimatedcenters of corneal curvature 1012. A CoR estimation module 724 maydetermine an estimated CoR based on a plurality of estimated centers ofcorneal curvature 1012 by fitting a surface to the estimated centers1012 and determining a region 1016 of convergence or intersection of aset of surface normal vectors 1014 normal to the surface. An estimatedCoR may be obtained from this region 1016, for example, the estimatedCoR may be at or within this region 1016.

Optionally, the CoR estimate(s) may be further checked using the eyetracking module 614. For example, as described in more detail below, ifthe CoR was moving with respect to the device during usage of a wearabledevice, new measurements of the cornea center 1012 may be tested bymeasuring the distance between the newly calculated cornea center 1012and the surface fitted to the set of calculated cornea centers 1012. Ifthe distance is too large, the eye tracking module 614 may pause eyetracking or switch to a different method of determining the CoR or adifferent eye tracking method. In some examples, the switch may betemporary until enough data is collected to reduce overall error.

Advantageously, the CoR extraction 1000 may employ one or moreassumptions. For example, the CoR extraction 1000 may assume that theglint extraction is accurate, that the geometry of the eye 1010 isknown, that the radius of the cornea (or two radii in the case of corneaastigmatism) is known, or that the data is collected during normal orrandom motion of the user's gaze.

L. EXAMPLE EYE TRACKING ENVIRONMENT

As discussed above, the CoR may be determined from a plurality ofestimated centers of corneal curvature 1012. For example, a surface maybe fit to the estimated centers of corneal curvature 1012 and aplurality of surface normal vectors 1014 normal to this surface may beobtained. A region 1016 of convergence of a set of these surface normalvectors 1014 may be identified. An estimated CoR may be obtained fromthis region of convergence 1016, for example, the estimated CoR may beat or within this region 1016.

To obtain the plurality of estimated centers of corneal curvature 1012,glints may be produced on the eye using illumination sources and imagedby a camera such as described above. FIG. 11 shows example images ofglints on any eye used by the eye tracking module for determining anestimated center of rotation. For example, as discussed above, awearable system may include an imaging system. The imaging system mayimage a user's eye 1110 to produce an eye image 1101. The wearablesystem may include one or more illumination source(s) 1102 that comprisespatially separate regions that output light. Accordingly, light fromthe illumination source(s) 1102 may produce one or more glint(s) 1104 onthe user's eye 1110 that are reflections of these spatially separateregions radiating light.

The imaging system of the wearable system may be part of an eye trackingassembly (for example, as shown in FIGS. 9A-E). The imaging system mayinclude one or more cameras. For example, the imaging system can includea single camera at a location 1106 in relation to a user's eye 1110. Inanother example, the imaging system can include multiple cameras thatmay be located at different locations in relation to the user's eye1110.

The illumination source(s) 1102 can include one or more light sourcessuch as light emitting diodes (LEDs). The illumination source(s) mayemit light in visible or non-visible light (for example, infrared (IR)light). For example, the illumination source(s) 1102 can be infrared(IR) LEDs. The illumination source(s) 1102 can be part of an eyetracking assembly (for example, as illustrated in FIGS. 9A-E).

The illuminations source(s) 1102 may produce one or more specularreflections 1104 on the cornea of a user's eye 1110. The specularreflections 1104 may also be referred to as glints. For example, theremay be two illumination sources (1102A, 1102B). The illuminationsource(s) may be configured to produce two or more discrete glints(1104A, 1104B) on the user's eye 1110. FIG. 11 shown an image of user'seye with the glints thereon. FIG. 11 also shows a view of the camera1106 (represented by the origin of the coordinate system) in comparisonto the location of the eye 1110 and the glints 1104A, 1104B thereon aswell as with respect to the illumination sources 1102A, 1102B at theirrelative locations.

M. EXAMPLE EXTRACTION OF A VECTOR ALONG WHICH A CORNEA CENTER IS LOCATEDUSING A SINGLE CAMERA

As discussed above, a camera at location 1106 may image glints 1104A,1104B on a user's eye 1110 that are produced by illumination sources1102A, 1102B. FIGS. 12A-D, a first plane 1220 that includes the locationof a glint 1104A, the camera capturing the image of the glint atlocation 1106, and the source of illumination 1102A producing the glintcan be determined. In particular, a module may determine a first plane1220 that includes a first illumination source 1102A and a first glint1104A. Similarly, as illustrated in FIGS. 13A-D, a second plane 1320that includes the locations of a glint 1104B, the camera capturing theimage of the glint at location 1106, and the source of illumination1102B producing the glint can be determined. In particular, the module716 may determine a second plane 1320 based on a second illuminationsource 1102B and a second glint 1104B. As illustrated in FIGS. 14A-C,the module 716 may determine the intersection between the first plane1220 and second plane 1320. The intersection between the first plane1220 and second plane 1320 may define a vector 1410 directed along wherethe cornea center is located. As shown in FIGS. 14A-C, this vector 1410may also extend along a direction that includes the location 1106 of thecamera.

In some implementations, the module 716 may determine a first plane 1220by determining a set of lines 1210, 1212, 1214 between a firstillumination source 1102A, a first glint 1104A, and camera location1106. As illustrated in FIG. 12A, the module 716 may determine a firstline 1210 extending between the camera location 1106 and the location inan image plane 1101A of a first glint 1104A that may be produced by afirst illumination source 1102A. As illustrated in FIG. 12B, the module716 may determine a second line 1212 extending between the cameralocation 1106 and the location of the illumination source 1102A thatproduced the first glint 1104A. As illustrated in FIG. 12C, the module716 may determine a third line 1214 cast between the location in theimage plane 1101A of the illumination sources 1102A and the first glint1104A. As illustrated in FIG. 12D, any two of these lines 1210, 1210,and 1214 may define a plane 1220 in which a cornea center may lie.

Similarly, in some implementations, the module 716 may determine asecond plane 1320 by determining a set of lines 1310, 1312, 1314 betweena second illumination source 1102B, a second glint 1104B, and cameralocation 1106. As illustrated in FIG. 13A, the module 716 may determinea first line 1310 extending between the camera location 1106 and thelocation in the image plane 1101A of a second glint 1104B that may beproduced by a second illumination source 1102B. As illustrated in FIG.13B, the module 716 may determine a second line 1313 extending betweenthe camera location 1106 and the location of the second illuminationsource 1102B that produced the second glint 1104A. As illustrated inFIG. 13C, the module 716 may determine a third line 1314 extendingbetween the location in the image plane 1101A of the second glint 1104Band the second illumination source 1102B. As illustrated in FIG. 13D,the lines 1310, 1310, and 1314 may define a plane 1320 in which a corneacenter may lie.

In some implementations, however, the first plane 1220 can be determineddirectly from the locations of the first illumination source 1102A andthe first glint 1104A, as well as the camera location 1106 withoutnecessarily separately defining the lines 1210, 1210, and 1214.Similarly, the second plane 1320 can be determined directly from thelocations of the second illumination source 1102B and the second glint1104B, as well as the camera location 1106 without necessarilyseparately defining the lines 1310, 1310, and 1314

The module 716 may identify an intersection between first and secondplanes 1220 and 1320. As illustrated in FIGS. 14A and 14B, theintersection of first plane 1220 and second plane 1320 may define avector 1410 with an origin at the camera location 1106 or otherwiseextending along a direction that may include the camera location. Asshown in FIG. 14C, the vector 1410 may point towards a cornea centerlocation.

The module 716 may repeat the estimation process multiple times togenerate one or more cornea vectors 1410. For example, the module 716may determine a first plane 1220 with which to define the vector basedon a first illumination source 1102A and a first glint 1104A withmultiple different camera locations 1106. The camera locations 1106 canbe varied in relation to a user's eye 1110 (for example, with respect toa distance to the user's eye 1110 or horizontal or vertical positionwith respect to the eye or any combination thereof) or with respect tothe location of an illumination source (1102A, 1102B). The module 716may determine vectors 1410 for one or more of the camera locations 1106.The module 716 may then determine the cornea center from an intersectionof two or more vectors as described above. If the two or more vectors donot intersect, then the cornea center may be interpolated or otherwiseextrapolated from the vector data. Additionally or alternatively, theeye tracking module 614 may collect and analyze more data to determinethe cornea center.

The module 716 may repeat the estimation process while varying one ormore parameters associated with an eye tracking environment 1100. Forexample, the module 716 may repeat the process with different cameralocations or for different gaze directions of the user's eye. The eyetracking module 614 may utilize gaze targets to ensure that a usermaintains their eye pose while a parameter is varied. For example, theeye tracking module 614 may estimate one or more vectors 1410 while theuser directs their gaze at the gaze targets while varying a parameter,such as the location 1106 of the camera or location of an illuminationsource 1102. Additionally or alternatively, the eye tracking module 614may estimate one or more vectors 1410 while the user naturally movestheir gaze during use of the wearable device. For example, the eyetracking module 614 may capture data associated with differentparameters during natural movement of the user's eye.

The repeated estimation process may result in multiple vectors 1410pointing to a cornea center associated with a particular eye pose. Themodule 716 may determine an intersection or region of convergence of themultiple vectors 1410 to generate an estimated center of cornealcurvature.

N. EXAMPLE EXTRACTION OF A VECTOR ALONG WHICH A CORNEA CENTER IS LOCATEDUSING MULTIPLE CAMERAS

In various implementations, multiple cameras may be employed to imagethe eye and images from the multiple cameras may be used to determinethe center of curvature of the cornea of that eye. In particular, themodule 716 may determine vectors (1510, 1530) along which the corneacenter may be located. FIGS. 15A-16C illustrate steps in an exampleprocess for determining such a vector with multiple cameras. Forexample, as illustrated in FIG. 15A, a first camera at a first location1506 may image glints 1504A, 1504B on a user's eye 1501 that areproduced by illumination sources 1502A, 1502B and a second camera at alocation 1526 may image glints 1524A, 1524B on a user's eye 1501 thatare produced by illumination sources 1522A, 1522B. The module 716 maydetermine a first vector 1510 based on data associated with the firstcamera at location 1506 and illumination sources 1502A, 1502B and maydetermine a second vector 1530 associated with the second camera atlocation 1526 illumination sources 1522A, 1522B. As illustrated in FIG.15B, the module 716 may estimate a cornea center 1520 by determining aconvergence or intersection between the first vector 1510 and secondvector 1530.

To obtain the first vectors 1510, the module 716 may identify a firstplane 1512 by determining a set of lines (not shown) between a firstillumination source 1502A, a first glint location 1504A in an imageplane 1503A, and a first camera at first location 1506. The module 716may determine a second plane 1514 by determining a set of lines (notshown) between a second illumination source 1502B, a second glintlocation 1504B in an image plane 1503A, and a second camera location1506. The module 716 may determine a vector 1510 by determining anintersection between these first and second planes 1512 and 1514. Theintersection of these planes 1512 and 1514 may define a vector 1510 withan origin at the camera location 1506 that point towards a cornea centerof curvature location.

In some implementations, however, the first plane 1512 can be determineddirectly from the locations of the first illumination source 1502A, thefirst glint 1504A, and the first camera 1106 without necessarilyseparately defining one or more lines. Similarly, the second plane 1514can be determined directly from the locations of the second illuminationsource 1502B, the second glint 1504B, and the first camera 1506 withoutnecessarily separately defining one or more lines.

A module 716 may similarly determine a first plane 1532 by determining aset of lines (not shown) between a first illumination source 1522A, afirst glint location 1524A in an image plane 1503B, and a first cameraat location 1526. The module 716 may determine a second plane 1534 bydetermining a set of lines (not shown) between a second illuminationsource 1522B, a second glint location 1524B in an image plane 1503B, andcamera location 1526. The module 716 may determine a second vector 1530by determining an intersection between these first and second planes1532 and 1534. The intersection of the planes 1532 and 1534 may define avector 1530 with an origin at the camera location 1526 that may pointtowards a cornea center of curvature location. In some implementations,however, the first plane 1532 can be determined directly from thelocations of the first illumination source 1522A, the first glint 1524A,and the second camera 1526 without necessarily separately defining oneor more lines. Similarly, the second plane 1534 can be determineddirectly from the locations of the second illumination source 1522B, thesecond glint 1524B, and the second camera 1526 without necessarilyseparately defining one or more lines.

As illustrated in FIG. 15B, the module 716 may determine a cornea centerof curvature location based on the these first and second vectors 1510and 1530. For example, the module 716 may determine a convergence orintersection 1520 of these vectors 1510 and 1530. The convergence orintersection 1520 may correspond to an approximate cornea centerlocation. If the vectors 1510 and 1530 do not intersect, then the corneacenter of curvature may be interpolated or otherwise extrapolated fromthe vector data. Additionally or alternatively, the eye tracking module614 may collect and analyze more data to determine the cornea center ofcurvature 1520.

FIGS. 16A-16C illustrate another example process for determining cornealcenter of curvature using multiple cameras. As illustrated in FIG. 16A,a wearable system may have a set of shared illumination sources 1602A,1602B that may be used with multiple eye cameras. The sharedillumination sources 1602A, 1602B may be in addition or in thealternative to a set of separate illumination sources associated withone or more cameras. The set of shared illumination sources 1602A, 1602Bmay produce glints 1604A, 1604B, 1604C, 1604D on a user's eye.

As illustrated in FIG. 16B, a module 716 may determine a set of planesusing the shared illumination sources 1602A, 1602B. For example, themodule 716 may determine a first plane 1630 by determining a set oflines (not shown) between a first illumination source 1602A, a firstglint location 1604A in an image plane 1503A, and a first camera atlocation 1506. The module 716 may determine a second plane 1632 bydetermining a set of lines (not shown) between a second illuminationsource 1602B, a second glint location 1604B in a first image plane1503A, and first camera location 1506.

In some implementations, however, the first plane 1630 can be determineddirectly from the locations of the first illumination source 1602A, thefirst glint 1604A in the first image plane 1503A, and the first camera1506 without necessarily separately defining one or more lines.Similarly, the second plane 1632 can be determined directly from thelocations of the second illumination source 1602B, the second glint1604B, and the first camera 1506 without necessarily separately definingone or more lines.

The module 716 may determine a different first plane 1634 by determininga set of lines (not shown) between the first illumination source 1602A,a first glint location 1604C in an image plane 1503B, and a secondcamera at location 1526. The module 716 may determine a separatedifferent plane 1636 by determining a set of lines (not shown) betweenthe second illumination source 1602B, a second glint location 1604D in asecond image plane 1503B, and second camera location 1526.

In some implementations, however, the different first plane 1634 can bedetermined directly from the locations of the first illumination source1602A, the first glint 1604C in the image plane 1503B, and the secondcamera 1526 without necessarily separately defining one or more lines.Similarly, the different second plane 1636 can be determined directlyfrom the locations of the second illumination source 1602B, the secondglint 1604D, and the second camera 1526 without necessarily separatelydefining one or more lines.

As illustrated in FIG. 16C, the module 614 may determine an intersectionbetween planes 1630 and 1632 to determine a vector 1610. Theintersection of the planes 1630 and 1632 may define the vector 1610 withan origin at the camera location 1506 that may point towards a corneacenter location. Similarly, the module 614 may determine an intersectionbetween planes 1634 and 1636 to determine a vector 1630. Theintersection of the planes 1634 and 1636 may define the vector 1630 withan origin at the camera location 1526 that may point towards a corneacenter location.

With continued reference to FIG. 16C, the module 716 may determine acornea center of curvature location based on the vectors 1610 and 1630.For example, the module 716 may determine a convergence or intersection1620 of the first and second vectors 1610 and 1630. The convergence orintersection 1620 may correspond to an approximate cornea center ofcurvature location. If the first and second vectors 1610 and 1630 do notintersect, then the cornea center of curvature may be interpolated orotherwise extrapolated from the vector data. Additionally oralternatively, the eye tracking module 614 may collect and analyze moredata to determine the cornea center of curvature.

The module 716 may repeat the estimation process for multiple gazedirections of the user's eye. For example, a wearable system may displayone or more gaze targets at which a user may direct their gaze. The eyetracking module 614 may estimate one or more vectors 1410 while the userdirects their gaze at the gaze targets. Additionally or alternatively,the eye tracking module 614 may estimate one or more vectors 1410 whilethe user naturally moves their gaze during use of the wearable device.For example, the eye tracking module 614 may capture data associatedwith different parameters during natural movement of the user's eye. Asdescribed below, the data captured at different eye poses or gazevectors of the user's eye may be used to calculate multiple corneacenters, which may be used by a CoR estimation module 724 to estimate aCoR.

O. ESTIMATING CENTER OF ROTATION

A Center of Rotation (CoR) estimation module 724 may determine anestimated center of rotation based on the estimated centers of cornealcurvature 1012. For example, the CoR estimation module 724 may fit asurface to one or more estimated cornea centers of curvature anddetermine a set of surface normal vectors normal to the fit surface. Thesurface normal vectors may converge or intersect at a point or regionthat may correspond to the estimated CoR.

To determine a surface, the module 614 may analyze multiple eye images.For example, a wearable system may image the user's eye 1501 (forexample, with the inward facing imaging system 462) while the user's eye1501 is in one or more eye poses. In some implementations, the module614 may prompt the one or more eye poses or gaze directions through thedisplay of gaze targets on a display of a wearable device. Additionallyor alternatively, the module 614 may collect data associated with one ormore eye poses that occur naturally during use of a wearable device.

As illustrated in FIGS. 17A and 17B, a module 614 may determine multiplecornea centers of curvature 1712 based on data collected by the wearablesystem while the user's eye is in one or more eye poses. For example,the module 614 may perform a cornea center of curvature estimationprocess as described above with one or more camera as part of module 716multiple times (e.g., for different gaze directions or eye poses of auser's eye 1501). The output of the cornea center estimation processesof module 716 may include multiple estimated cornea centers of curvature1712.

The multiple cornea centers of curvature 1712 may be situated within aregion 1710 of three-dimensional (3D) space. The region 1710 may fallwithin the corneal sphere 1022. Without subscribing to any particularscientific theory, the multiple cornea centers of curvature 1712 mayapproximately align within the region 1710 according to a shape of thecorneal curvature 1018. For example, the multiple cornea centers ofcurvature 1712 may align within the region 1710 so as to outline a shapesubstantially parallel to or substantially the same as the shape of thecornea 1020. In cases where the cornea is substantially spherical, themultiple cornea centers 1712 may approximately follow a cornea curvature1018 at a distance approximately equivalent to the radius of the cornea.In cases of astigmatism (or where the cornea is not substantiallyspherical), the multiple cornea centers 1712 may approximately follow acornea curvature 1018 at distances approximately equivalent to one ormore radii of the corneal geometry.

In various implementations, the module 614 may determine if the multiplecornea centers 1712 fall within a determined margin of an expecteddistance to the center of the corneal sphere 1022 from a surface of thecornea 1022. For example, a corneal sphere 1022 may be spherical orastigmatic (e.g., have a geometry other than a spherical shape). Anexpected distance may correspond to a distance to a center of thecorneal sphere 1022 geometry. For example, where the corneal geometry isspherical, the expected distance may be the radius of the corneal sphere1022. If a cornea center 1712 falls outside of the determined margin,the module 614 may reduce the contribution of the outlier in furtheranalysis. For example, the module 614 may exclude the outlying datapoint from further analysis. Additionally or alternatively, if athreshold number of cornea centers 1712 falls outside of the determinedmargin, the module 614 may stop analysis until further data is acquiredor switch to a different method of determining center of rotation.

As shown in FIG. 17B, a module 724 may fit a 3D surface 1714 to themultiple cornea centers 1712. The module 724 may fit a 3D surface, forexample, using regression analysis. The module 724 may utilize asuitable surface or curve fitting technique to determine the fit. Themodule 724 may use, for example, polynomial regression to fit the corneacenters 1712 to a low order polynomial 3D surface 1714. In anotherexample, the module 724 may apply a geometric fit to the cornea centers1712 (e.g. a total least squares fit). In some examples, the surface1714 may have a similar curvature to the corneal curvature 1018. Inother examples, the surface 1714 may have a shape different from thecorneal curvature 1018.

The module 724 may determine a set of surface normal vectors that arenormal to the surface 1714. FIG. 18A illustrates an example calculation1800 of a CoR (or eye ball center “EBC”) using surface normal vectors1814. For example, a module 716 may determine a set of estimated corneacenters 1812. The module 724 may fit a surface 1714 (as shown in FIG.17B) to the estimated cornea centers 1812. The module 724 may thendetermine one or more surface normal vectors 1814 that are normal to thesurface 1714. The surface normal vectors 1814 may originate from theestimated centers of corneal curvature 1812. For example, the module 724may determine a surface normal vector 1814 for each estimated center ofcorneal curvature 1812 used to determine the surface 1714. Less surfacenormal may be used in certain implementations. Additionally oralternatively, the surface normal vectors 1814 may originate from otherpoints on the surface 1714.

The module 724 may determine a region of convergence 1802 of the surfacenormal vectors 1814. For example, as illustrated in inset 1801 of FIG.18A, some or all of the surface normal vectors 1814 may converge orintersect in a region 1802 of 3D space. The region of 1802 of 3D spacemay be a point of intersection or a volume of 3D space (for example,volume 1920 in FIGS. 19C and 19D) where the normal vectors intersectand/or converge. The volume of 3D space may be centered around a medianpoint of intersection or convergence of the surface normal vectors 1814.The volume of 3D space may be large enough to encompass a majority ofintersection points.

The region of convergence 1802 can include different areas ofconvergence or intersection corresponding to different gaze directionsor eye poses. For example, the region of convergence 1802 can include asub-region 1820 corresponding to a first gaze direction (e.g., a bottomgaze) and a sub-region 1822 corresponding to a second gaze direction(e.g., a top gaze). In some examples, the sub-regions 1820, 1822 cancorrespond to an approximated CoR associated with a region of thedisplay of a wearable device. For example, a first sub-region 1820 cancorrespond to an upper region of the display and a second sub-region1822 can correspond to a lower region of the display.

The module 724 may determine a CoR by analyzing the region ofconvergence 1802. For example, the module 724 may determine a CoR bydetermining the mode or median of convergence or intersection points ofthe vectors 1814. Additionally or alternatively, the module 724 maydetermine a CoR by first determining gaze based convergence orintersection points, such as the mode or median of convergence orintersection points of vectors 1814 in sub-regions 1820, 1822, and thendetermining a mode or median based on those gaze based convergence orintersection points. Additionally or alternatively, the module 724 mayperform a different analysis of the convergence or intersection pointsto determine a CoR. For example, the module 724 may utilize a machinelearning algorithm to determine a CoR.

In some examples, variation in the calculated cornea centers ofcurvature may result in a broader region of convergence 1824 as opposedto a single point of intersection. FIG. 18B illustrates an example CoRcalculation 1803 with a region 1824. For example, calculated corneacenters of curvature 1832 may be noisy with respect to a fitted 3Dsurface 1830. The noisy cornea centers 1832 may result in a region 1824in which a CoR or eye ball center (EBC) is likely to be based on theintersection of vectors (not shown) with an origin at the cornea centers1832. In some implementations, the module 614 may use the region 1824 incalculating a gaze direction. For example, the module 614 may determinea CoR as the center of the region 1824 or some other location within oron or otherwise based on the region 1824.

In various implementations, the module 724 may select a portion ofestimated cornea centers 1910 to determine a CoR. FIGS. 19A-1 and 19A-2illustrate an example surface 1912 fit to a portion of estimated corneacenters 1910 that may be selected using a data reduction process. FIGS.19B-1 and 19B-2 shows example vectors 1916 that may be normal to thesurface 1912. The vectors 1916 may originate at the selected estimatedcornea centers 1910. FIGS. 19C-1 and 19C-2 illustrate an estimated CoRregion 1920 based on points of or a region convergence or intersectionof the vectors 1916. As shown in FIGS. 19D-1 and 19D-2, where the module724 does not select cornea centers 1910 to fit a surface 1912, many ofthe vectors 1922 may not converge or intersect within the region 1920.

In various implementations, the module 724 may select estimated corneacenters 1910 based on a determined region of convergence of normalvectors 1916. For example, the module 724 may determine a large regionin which the normal vectors 1922 intersect. In some implementations, ifthe large region has a volume greater than a threshold volume, themodule 724 may determine a smaller set of the cornea centers 1910 withwhich to determine the CoR. In certain implementations, the thresholdvolume can include a suitable volume for determining a CoR associatedwith a threshold accuracy of gaze tracking based on that CoR. Forexample, a volume of 30 percent of the volume of the user's eye could beassociated with an 80% decrease in accuracy in gaze tracking. Where thedetermined volume is greater than the threshold volume, the module 724may select a smaller set of cornea centers 1910 based on any number ofsuitable data selection criteria, as described below.

Additionally or alternatively, the module 724 may select estimatedcornea centers 1910 for analysis using any number of data reductionprocesses, such as a machine learning algorithm or a filtration process.For example, the module 724 may filter the data to eliminate outliers.The filter may include determining a confidence score associated with agiven cornea center 1910 and selecting cornea centers 1910 based ontheir confidence scores. In some examples, the confidence scores may bedetermined based on a cornea center(s) of curvature 1910 deviation froma secondary calculation or determination of the cornea center(s) ofcurvature 1910 or surface 1912. In some examples, confidence scores maybe based on the location of the cornea centers of curvature 1910 inrelation to a fit surface 1912 (e.g., a deviation of the cornea centers1910 from the fit surface 1912). In some examples, the confidence scoresmay be determined based on a calculated error in the glint extractionutilized to determine the cornea centers of curvature 1910. For example,the glint extraction may have a high error if there is error in the eyeimage(s) analyzed to extract the glint (e.g., due to blur, obstructionin the image, distortion, or other sources of noise).

P. EXAMPLE APPLICATION OF CENTER OF CORNEAL CURVATURE OF ROTATIONEXTRACTION

FIG. 20 illustrates an example center of rotation extraction process2100 that may be implemented by eye tracking module 614. For example,the center of rotation extraction process 2100 can include one or morecornea center estimation processes 2108, a fitting block 2116, a vectordetermination block 2118, a converges or intersection determinationblock 2120, and a center of rotation determination block 2122.

The module 614 can perform a number of blocks as part of the one or morecornea center estimation processes 2108. For example, a cornea centerestimation process 2108 can include an image receiving block 2110, aglint determination block 2112, and a cornea center determination block2114.

At an image receiving block 2110, the module 614 can receive one or moreimages of a user's eye. The images can be obtained from an imagingsystem associated with a wearable device worn by the user. For example,the wearable device can be a head mounted display that includes a lefteyepiece 2010A and a right eyepiece 2010B with imaging systems thatinclude inward-facing cameras 2014, 2016, 2018, and 2020 as illustratedin FIGS. 9A-9D. The module 614 can optionally analyze the images forquality. For example, the module 614 can determine if the images pass aquality threshold. The threshold can include metrics for quality of theimage relating to blur, obstruction, unwanted glints, or other qualitymetrics that may affect the accuracy of the center of rotation analysis.If the module 614 determines that the image passes the image qualitythreshold, the module 614 may use the image in further analysis.

At a glint determination block 2112, the module 614 may analyze theimage(s) received from block 2110 to determine a location of one or moreglints within the image(s). As described above with reference to FIGS.12A-16C, the glint locations can correspond to a location of one or moreglints produced by one or more illumination sources in an image plane.Additionally or alternatively, the glint locations can correspond to alocation of one or more glints produced by one or more illuminationsources in the coordinate frame of the user's eye. Glints can also beobtained for different camera and/or different camera locations.

At a cornea center determination block 2114, the module 614 can analyzethe glint locations to determine an estimated cornea center ofcurvature. As described above with reference to FIGS. 12A-16C, thedetermination can involve determining a vector along which the corneacenter of curvature is located based on glint, illumination source, andcamera locations. The determination can also involve determining anestimated cornea center of curvature based on an intersection locationof one or more of those vectors.

Additionally or alternatively, the module 614 can perform one or moreblocks multiple times. For example, the module 614 may perform blocks2110, 2112, and 2114 multiple times. For example, the module 614 mayperform 2112 and 2114 multiple times for each eye image or set of eyeimages from block 2110 in order to calculate one or more cornea centersof curvature. In another example, the module 614 may perform blocks2110, 2112, and 2114 for multiple eye poses or conditions. For example,the module 614 can receive images of a user's eye(s) in different eyeposes or different gaze directions. Additionally or alternatively, themodule 614 can receive images of a user's eye(s) with different cameraconditions, such as camera distance from the user's eye, vertical orhorizontal location with respect to the user's eye, or any combinationthereof, which may provide different camera perspectives and/or fordifferent cameras having different locations and/or perspectives. Asdescribed above, a wearable device can prompt the user to engage indifferent eye poses by, for example, causing the display of gaze targetsin different regions of the display. For example, the wearable devicecan display five gaze targets corresponding to an upper center region ofa display, a lower center region of the display, a central region of thedisplay, a left of center region of the display, and a right of centerregion of the display. The five gaze targets may correspond to fivedifferent eye poses of the user. Additionally or alternatively, thewearable system may capture different eye poses that occur during anatural movement of the user's eyes during use of the wearable system.

The module 614 may continue to collect data until a threshold criteriais met. For example, the threshold criteria can include a margin oferror, a number of data points, or a minimum, threshold, or targetdiversity of eye poses. In some examples, the margin of error cancorrespond to a minimum, threshold, or target number of calculatedcornea centers of curvature, a minimum, threshold, or target error levelis achieved in a calculated center of rotation or deviation of corneacenters of curvature from a fitted surface, some combination thereof orthe like. Other approaches are possible.

At block 2116, the module 614 may fit a surface to the one or morecornea centers output from processes 2108. As described above withreference to FIGS. 17A and 17B, the module 614 may perform a regressionanalysis to generate a fitted surface. For example, the module 614 mayperform a polynomial regression to generate a low order polynomial 3Dsurface to the cornea centers. Other techniques, however, may be used.

At block 2118, the module 614 may determine surface normal vectors fromthe surface fit at block 2116. As described above with reference to FIG.18A, the module 614 may determine surface normal vectors that are normalto the fitted surface that originate at or go through the cornea centersof curvature. Additionally or alternatively, the module 614 maydetermine surface normal vectors originating from any point on thefitted surface. Other approaches are also possible.

At block 2120, the module 614 may determine a region of convergence ofthe surface normal vectors determined at block 2118. As described abovewith reference to FIGS. 19A-19E, the module 614 may determine a point orregion of convergence of the surface normal vectors. The region ofconvergence may be a volume of space in which a substantial portion ofthe surface normal vectors converge and/or intersect. The point orregion of convergence may approximately correspond to or assist inestimating a center of rotation of the user's eye.

At block 2122, the module 614 may determine a center of rotation basedon the determined region of convergence from block 2120. The center ofrotation may for example, be at, within, or on the region ofconvergence. Other locations may also be determined for the center ofrotation based on the region of convergence. In some implementations, asdescribed above, the module 614 may analyze the region of convergencefor a threshold criteria (e.g., an error). If the module 614 determinesthat the region of convergence does not meet a threshold criteria (e.g.,for error and/or volume), the module 614 may not output a center ofrotation. If the module 614 determines that the region of convergencemeets the threshold criteria, then the module 614 may determine that thecenter of rotation is a center of the region of convergence.

Q. EXAMPLE EYE TRACKING PROCESS USING CENTER OF CORNEAL CURVATURE FORCENTER OF ROTATION EXTRACTION

FIG. 21 illustrates an example eye tracking process 2200 that may use aprocess 2100 of determining the center of corneal curvature for centerof rotation extraction (e.g., as described above with reference to FIG.20). The process 2200 in this example can include a center of rotationdetermination block 2210, an error determination block 2212, a thresholddetermination block 2214, an alternative eye tracking block 2216, and acorneal eye tracking block 2218.

At the center of rotation determination block 2210, the module 614 candetermine a center of rotation using corneal data. For example, themodule 614 may determine a center of rotation using a process 2100described above with reference to FIG. 21. At the error determinationblock 2212, the module 614 can determine an error associated with thecenter of rotation from block 2210. At the block 2214, the module 614can analyze the error from block 2212 to determine if it exceeds athreshold error value. In some implementations, the threshold errorvalue can correspond to a value associated with a deviation (e.g.,heighted or threshold deviation) of the center of rotation from anexpected value. In some implementations, the expected value can includea location of the center of rotation based on a different center ofrotation determination process, an expected center of rotation based onan eye geometry, an average center of rotation across a population ofusers, or another suitable center of rotation value. Other thresholdvalues may be used. If the error exceeds the threshold, the module 614may utilize an alternate eye tracking or center of rotation estimationmethod at block 2216. If the error does not exceed the threshold, thenthe module 614 may utilize the calculated center of rotation from block2210 at block 2218.

R. EXAMPLES FOR WHEN EYE TRACKING IS UNAVAILABLE

In some embodiments, eye tracking may not be provided or may betemporarily unavailable. As examples, the eye tracking camera 324 orlight sources 326 may be obscured, damaged, or disabled by a user, theenvironmental lighting conditions may make eye tracking prohibitivelydifficult, the wearable system may be improperly fitted in a manner thatprevents eye tracking, the user may be squinting or have eyes that arenot easily tracked, etc. At such times, the wearable system may beconfigured to fall back upon various strategies for positioning therender camera and selecting depth planes in the absence of eye trackingdata.

For example, with respect to the render camera, the wearable system mayposition the render camera to a default position if the user's pupilsare not detected for longer than a predetermined threshold, such as afew seconds or longer than a typical blink. The wearable system maypossibly move the render camera to the default position in a smoothmovement, which may, e.g., follow an over-damped oscillator model. Insome implementations, the default position may be determined as part ofa calibration process of the wearable system to a particular user.However, the default position may be a user's left and right eyes'centers of rotation. These are merely illustrative examples.

S. COMPUTER VISION TO DETECT OBJECTS IN AMBIENT ENVIRONMENT

As discussed above, the display system may be configured to detectobjects in or properties of the environment surrounding the user. Thedetection may be accomplished using a variety of techniques, includingvarious environmental sensors (e.g., cameras, audio sensors, temperaturesensors, etc.), as discussed herein.

In some embodiments, objects present in the environment may be detectedusing computer vision techniques. For example, as disclosed herein, thedisplay system's forward-facing camera may be configured to image theambient environment and the display system may be configured to performimage analysis on the images to determine the presence of objects in theambient environment. The display system may analyze the images acquiredby the outward-facing imaging system to perform scene reconstruction,event detection, video tracking, object recognition, object poseestimation, learning, indexing, motion estimation, or image restoration,etc. As other examples, the display system may be configured to performface and/or eye recognition to determine the presence and location offaces and/or human eyes in the user's field of view. One or morecomputer vision algorithms may be used to perform these tasks.Non-limiting examples of computer vision algorithms include:Scale-invariant feature transform (SIFT), speeded up robust features(SURF), oriented FAST and rotated BRIEF (ORB), binary robust invariantscalable keypoints (BRISK), fast retina keypoint (FREAK), Viola-Jonesalgorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunkalgorithm, Mean-shift algorithm, visual simultaneous location andmapping (vSLAM) techniques, a sequential Bayesian estimator (e.g.,Kalman filter, extended Kalman filter, etc.), bundle adjustment,Adaptive thresholding (and other thresholding techniques), IterativeClosest Point (ICP), Semi Global Matching (SGM), Semi Global BlockMatching (SGBM), Feature Point Histograms, various machine learningalgorithms (such as e.g., support vector machine, k-nearest neighborsalgorithm, Naive Bayes, neural network (including convolutional or deepneural networks), or other supervised/unsupervised models, etc.), and soforth.

One or more of these computer vision techniques may also be usedtogether with data acquired from other environmental sensors (such as,e.g., microphone) to detect and determine various properties of theobjects detected by the sensors.

As discussed herein, the objects in the ambient environment may bedetected based on one or more criteria. When the display system detectsthe presence or absence of the criteria in the ambient environment usinga computer vision algorithm or using data received from one or moresensor assemblies (which may or may not be part of the display system),the display system may then signal the presence of the object.

T. MACHINE LEARNING

A variety of machine learning algorithms may be used to learn toidentify the presence of objects in the ambient environment. Oncetrained, the machine learning algorithms may be stored by the displaysystem. Some examples of machine learning algorithms may includesupervised or non-supervised machine learning algorithms, includingregression algorithms (such as, for example, Ordinary Least SquaresRegression), instance-based algorithms (such as, for example, LearningVector Quantization), decision tree algorithms (such as, for example,classification and regression trees), Bayesian algorithms (such as, forexample, Naive Bayes), clustering algorithms (such as, for example,k-means clustering), association rule learning algorithms (such as, forexample, a-priori algorithms), artificial neural network algorithms(such as, for example, Perceptron), deep learning algorithms (such as,for example, Deep Boltzmann Machine, or deep neural network),dimensionality reduction algorithms (such as, for example, PrincipalComponent Analysis), ensemble algorithms (such as, for example, StackedGeneralization), and/or other machine learning algorithms. In someembodiments, individual models may be customized for individual datasets. For example, the wearable device may generate or store a basemodel. The base model may be used as a starting point to generateadditional models specific to a data type (e.g., a particular user), adata set (e.g., a set of additional images obtained), conditionalsituations, or other variations. In some embodiments, the display systemmay be configured to utilize a plurality of techniques to generatemodels for analysis of the aggregated data. Other techniques may includeusing pre-defined thresholds or data values.

The criteria for detecting an object may include one or more thresholdconditions. If the analysis of the data acquired by the environmentalsensor indicates that a threshold condition is passed, the displaysystem may provide a signal indicating the detection the presence of theobject in the ambient environment. The threshold condition may involve aquantitative and/or qualitative measure. For example, the thresholdcondition may include a score or a percentage associated with thelikelihood of the reflection and/or object being present in theenvironment. The display system may compare the score calculated fromthe environmental sensor's data with the threshold score. If the scoreis higher than the threshold level, the display system may detect thepresence of the reflection and/or object. In some other embodiments, thedisplay system may signal the presence of the object in the environmentif the score is lower than the threshold. In some embodiments, thethreshold condition may be determined based on the user's emotionalstate and/or the user's interactions with the ambient environment.

In some embodiments, the threshold conditions, the machine learningalgorithms, or the computer vision algorithms may be specialized for aspecific context. For example, in a diagnostic context, the computervision algorithm may be specialized to detect certain responses to thestimulus. As another example, the display system may execute facialrecognition algorithms and/or event tracing algorithms to sense theuser's reaction to a stimulus, as discussed herein.

It will be appreciated that each of the processes, methods, andalgorithms described herein and/or depicted in the figures may beembodied in, and fully or partially automated by, code modules executedby one or more physical computing systems, hardware computer processors,application-specific circuitry, and/or electronic hardware configured toexecute specific and particular computer instructions. For example,computing systems may include general purpose computers (e.g., servers)programmed with specific computer instructions or special purposecomputers, special purpose circuitry, and so forth. A code module may becompiled and linked into an executable program, installed in a dynamiclink library, or may be written in an interpreted programming language.In some embodiments, particular operations and methods may be performedby circuitry that is specific to a given function.

Further, certain embodiments of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, a video mayinclude many frames, with each frame having millions of pixels, andspecifically programmed computer hardware is necessary to process thevideo data to provide a desired image processing task or application ina commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. In some embodiments,the non-transitory computer-readable medium may be part of one or moreof the local processing and data module (140), the remote processingmodule (150), and remote data repository (160). The methods and modules(or data) may also be transmitted as generated data signals (e.g., aspart of a carrier wave or other analog or digital propagated signal) ona variety of computer-readable transmission mediums, includingwireless-based and wired/cable-based mediums, and may take a variety offorms (e.g., as part of a single or multiplexed analog signal, or asmultiple discrete digital packets or frames). The results of thedisclosed processes or process steps may be stored, persistently orotherwise, in any type of non-transitory, tangible computer storage ormay be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities may be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto may be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe embodiments described herein is for illustrative purposes and shouldnot be understood as requiring such separation in all embodiments. Itshould be understood that the described program components, methods, andsystems may generally be integrated together in a single computerproduct or packaged into multiple computer products.

U. OTHER CONSIDERATIONS

Each of the processes, methods, and algorithms described herein and/ordepicted in the attached figures may be embodied in, and fully orpartially automated by, code modules executed by one or more physicalcomputing systems, hardware computer processors, application-specificcircuitry, and/or electronic hardware configured to execute specific andparticular computer instructions. For example, computing systems caninclude general purpose computers (e.g., servers) programmed withspecific computer instructions or special purpose computers, specialpurpose circuitry, and so forth. A code module may be compiled andlinked into an executable program, installed in a dynamic link library,or may be written in an interpreted programming language. In someimplementations, particular operations and methods may be performed bycircuitry that is specific to a given function.

Further, certain implementations of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, animationsor video may include many frames, with each frame having millions ofpixels, and specifically programmed computer hardware is necessary toprocess the video data to provide a desired image processing task orapplication in a commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. The methods andmodules (or data) may also be transmitted as generated data signals(e.g., as part of a carrier wave or other analog or digital propagatedsignal) on a variety of computer-readable transmission mediums,including wireless-based and wired/cable-based mediums, and may take avariety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). The resultsof the disclosed processes or process steps may be stored, persistentlyor otherwise, in any type of non-transitory, tangible computer storageor may be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities can be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto can be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe implementations described herein is for illustrative purposes andshould not be understood as requiring such separation in allimplementations. It should be understood that the described programcomponents, methods, and systems can generally be integrated together ina single computer product or packaged into multiple computer products.Many implementation variations are possible.

The processes, methods, and systems may be implemented in a network (ordistributed) computing environment. Network environments includeenterprise-wide computer networks, intranets, local area networks (LAN),wide area networks (WAN), personal area networks (PAN), cloud computingnetworks, crowd-sourced computing networks, the Internet, and the WorldWide Web. The network may be a wired or a wireless network or any othertype of communication network.

The systems and methods of the disclosure each have several innovativeaspects, no single one of which is solely responsible or required forthe desirable attributes disclosed herein. The various features andprocesses described above may be used independently of one another, ormay be combined in various ways. All possible combinations andsubcombinations are intended to fall within the scope of thisdisclosure. Various modifications to the implementations described inthis disclosure may be readily apparent to those skilled in the art, andthe generic principles defined herein may be applied to otherimplementations without departing from the spirit or scope of thisdisclosure. Thus, the claims are not intended to be limited to theimplementations shown herein, but are to be accorded the widest scopeconsistent with this disclosure, the principles and the novel featuresdisclosed herein.

Certain features that are described in this specification in the contextof separate implementations also can be implemented in combination in asingle implementation. Conversely, various features that are describedin the context of a single implementation also can be implemented inmultiple implementations separately or in any suitable subcombination.Moreover, although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination. No single feature orgroup of features is necessary or indispensable to each and everyembodiment.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list. In addition, thearticles “a,” “an,” and “the” as used in this application and theappended claims are to be construed to mean “one or more” or “at leastone” unless specified otherwise.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: A, B, or C” is intended to cover: A, B, C,A and B, A and C, B and C, and A, B, and C. Conjunctive language such asthe phrase “at least one of X, Y and Z,” unless specifically statedotherwise, is otherwise understood with the context as used in generalto convey that an item, term, etc. may be at least one of X, Y or Z.Thus, such conjunctive language is not generally intended to imply thatcertain embodiments require at least one of X, at least one of Y and atleast one of Z to each be present.

Similarly, while operations may be depicted in the drawings in aparticular order, it is to be recognized that such operations need notbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Further, the drawings may schematically depict one more exampleprocesses in the form of a flowchart. However, other operations that arenot depicted can be incorporated in the example methods and processesthat are schematically illustrated. For example, one or more additionaloperations can be performed before, after, simultaneously, or betweenany of the illustrated operations. Additionally, the operations may berearranged or reordered in other implementations. In certaincircumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system components in theimplementations described above should not be understood as requiringsuch separation in all implementations, and it should be understood thatthe described program components and systems can generally be integratedtogether in a single software product or packaged into multiple softwareproducts. Additionally, other implementations are within the scope ofthe following claims. In some cases, the actions recited in the claimscan be performed in a different order and still achieve desirableresults.

V. EXAMPLES

Examples of a user to display virtual image content in a vision field ofsaid user are described herein such as the examples enumerated below:

Example 1: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising: a frame configured to be supported on ahead of the user; a head-mounted display disposed on the frame, saiddisplay configured to project light into said user's eye to displayvirtual image content to the user's vision field; first and second eyetracking cameras configured to image the user's eye; a plurality oflight emitters; and processing electronics in communication with thedisplay and the first and second eye tracking cameras, the processingelectronics configured to: receive images of the user's eye captured bythe first and second eye tracking cameras, glint reflections of thedifferent light emitters observable in said images of the eye capturedby the first and second eye tracking cameras; and estimate a location ofsaid center of corneal curvature of the user's eye based on the locationof the glint reflections in said images produced by both said first andsecond eye tracking camera and based on the location of both the firstand second eye tracking cameras and the locations of the emitters thatproduced said respective glint reflections.

Example 2: The display system of Example 1, wherein said processingelectronics is configured to: based on the location of the glintreflections in one or more images produced by said first eye trackingcamera and based on the location of the first eye tracking camera andthe location of the emitters that produced said glint reflections,determine a first direction toward the center of corneal curvature ofthe user's eye; and based on the location of the glint reflections inone or more images produced by said second eye tracking camera and basedon the location of the second eye tracking camera and the location ofthe emitters that produced said glint reflections, determine a seconddirection toward the center of corneal curvature of the user's eye.

Example 3: The display system of Example 2, wherein said processingelectronics is configured to determine the first direction by: defininga first plane that includes the first eye tracking camera, a location ofa first glint reflection and a location of the light emittercorresponding to said first glint reflection; defining a second planethat includes the first eye tracking camera, a location of a secondglint reflection and a location of the light emitter corresponding tosaid second glint reflection; and determining a region of convergence ofthe first plane and the second plane, the region of convergenceextending along the first direction.

Example 4: The display system of Example 3, said processing electronicsare configured to determine the second direction by: defining a thirdplane that includes the second eye tracking camera, the location of athird glint reflection, and a location of the light emittercorresponding to said third glint reflection; defining a fourth planethat includes the second eye tracking camera, the location of a fourthglint reflection, and a location of the light emitter corresponding tosaid fourth glint reflection; and determining a region of convergence ofthe third plane and the fourth plane, the region of convergenceextending along the second direction.

Example 5: The display system of any of the examples above, wherein saidprocessing electronics is configured to estimate a location of saidcenter of corneal curvature of the user's eye based on said first andsecond directions toward the center of the corneal curvature of theuser's eye.

Example 6: The display system of any of the examples above, wherein saidprocessing electronics is configured to: determine said first directionalong which the center of corneal curvature of the user's eye isestimated to be located based on at least one first image received fromthe first eye tracking camera; and determine said second direction alongwhich the center of corneal curvature of the user's eye is estimated tobe located based on at least one second image received from the secondeye tracking camera, said first and second directions converging towarda region.

Example 7: The display system of any of the examples above, wherein saidprocessing electronics is configured to: obtain an estimate of a centerof corneal curvature of the user's eye based on the convergence of thefirst and second directions.

Example 8: The display system of any of the examples above, wherein saidprocessing electronics is configured to estimate a location of saidcenter of corneal curvature of the user's eye by identifying a region ofconvergence of said first and second directions toward the center of thecorneal curvature of the user's eye.

Example 9: The display system of any of the examples above, wherein saidprocessing electronics is configured to obtain an estimate of a centerof rotation of the user's eye based on multiple determinations of thecenter of corneal curvature of the user's eye for different eye poses.

Example 10: The display system of any of the examples above, whereinsaid processing electronics is configured to determine a locus of pointscorresponding to estimates of the center of corneal curvature of theuser's eye for different eye poses.

Example 11: The display system of Example 10, wherein said processingelectronics is configured to obtain an estimate of a center of rotationof the user's eye based on said locus of points corresponding toestimates of the center of corneal curvature of the user's eye fordifferent eye poses.

Example 12: The display system of Examples 10 or 11, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye.

Example 13: The display system of Examples 10 or 11, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye by estimating a center of curvature of said surface.

Example 14: The display system of Examples 10 or 11, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye by determining a region where a plurality of normalsto said surface converge.

Example 15: The display system of any of Examples 12, 13, or 14, whereinsaid processing electronics is configured to fit said surface to saidlocus of points to obtain said surface.

Example 16: The display system of any of the examples above, whereinsaid processing electronics is configured to use a render camera torender virtual images to be presented to the eye of the user, saidrender camera having a position determined by said center of rotation.

Example 17: The display system of any of the examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content to the user's vision field at differentamounts of at least one of divergence and collimation and thus thedisplayed virtual image content appears to originate from differentdepths at different periods of time.

Example 18: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising: a frame configured to be supported on ahead of the user; a head-mounted display disposed on the frame, saiddisplay configured to project light into said user's eye to displayvirtual image content to the user's vision field; first and second eyetracking cameras configured to image the user's eye; a plurality oflight emitters; and processing electronics in communication with thedisplay and the first and second eye tracking cameras, the processingelectronics configured to: receive images of the user's eye captured bythe first and second eye tracking cameras, glint reflections of thedifferent light emitters observable in said images of the eye capturedby the first and second eye tracking cameras; and estimate a location ofsaid center of rotation of the user's eye based on the location of theglint reflections in said images produced by both said first and secondeye tracking camera and based on said the location of both the first andsecond eye tracking cameras and the locations of the emitters thatproduced said glint reflections for multiple eye poses.

Example 19: The system of example 18, wherein to obtain an estimate ofthe center of rotation of said eye, the processing electronics areconfigured to: determine a plurality of estimates of the center ofcorneal curvature of the user's eye based a plurality of glintreflections for multiple eye poses; and determine the estimate of thecenter of rotation of the user's eye based on the plurality of estimatesof the center of corneal curvature of the user's eye for said multipleeye poses.

Example 20: The system of example 19, wherein to determine saidplurality of estimates of the corneal curvature of the user's eye, theprocessing electronics are configured to: determine a first directiontoward the center of corneal curvature based on the respective locationsof at least a portion of said plurality of emitters and a first cameraof the eye tracking cameras; determine a second direction toward thecenter of corneal curvature based on the respective locations of atleast a portion of said plurality of emitters and a second camera of theeye tracking cameras; and determine an estimate of the center of cornealcurvature of the user's eye based on said the first and seconddirections.

Example 21: The display system of Example 20, wherein said processingelectronics is configured to determine the first direction by: defininga first plane that includes the first eye tracking camera, a location ofa first glint reflection and a location of the light emittercorresponding to said first glint reflection; defining a second planethat includes the first eye tracking camera, a location of a secondglint reflection and a location of the light emitter corresponding tosaid second glint reflection; and determining a region of convergence ofthe first plane and the second plane, the region of convergenceextending along the first direction.

Example 22: The display system of Example 21, said processingelectronics are configured to determine the second direction by:defining a third plane that includes the second eye tracking camera, thelocation of a third glint reflection, and a location of the lightemitter corresponding to said third glint reflection; defining a fourthplane that includes the second eye tracking camera, the location of afourth glint reflection, and a location of the light emittercorresponding to said fourth glint reflection; and determining a regionof convergence of the third plane and the fourth plane, the region ofconvergence extending along the second direction.

Example 23: The system of any of Examples 20-22, wherein to determinesaid plurality of estimates of the corneal curvature of the user's eye,the processing electronics are configured to: determine a region ofconvergence between the first direction and second direction todetermine an estimate of the center of corneal curvature of the user'seye.

Example 24: The system of any of Examples 19-23, wherein to obtain anestimate of the center of rotation of said eye, the processingelectronics are configured to: generate a three-dimensional surfaceassociated with the plurality of estimates of the center of the cornealcurvature; and determine the estimate of the center of rotation of theuser's eye based on the three-dimensional surface.

Example 25: The system of example 24, wherein to generate athree-dimensional surface associated with the plurality of estimates ofthe center of the corneal curvature, the processing electronics areconfigured to fit a surface to the plurality of estimates of the centerof the corneal curvature.

Example 26: The system of example 24, wherein to generate athree-dimensional surface associated with the plurality of estimates ofthe center of the corneal curvature, the processing electronics areconfigured to fit a sphere to the plurality of estimates of the centerof the corneal curvature.

Example 27: The system of any of Example 24-26, wherein to determine theestimate of the center of rotation of the user's eye, the processingelectronics are configured to: determine two or more normals to thethree-dimensional surface; and determine a region of convergence of thetwo or more normals, wherein the region of convergence comprises theestimate of the center of rotation of the user's eye.

Example 28: The system of any of Examples 21-27, wherein the one or moreimages of the user's eye comprise one or more images associated withdifferent gaze vectors of the user's eye.

Example 29: The system of any of Examples 21-28, wherein the processingelectronics are configured to map the cornea of the user's eye using agaze target.

Example 30: The display system of any of Examples 18-29, wherein saidprocessing electronics is configured to use a render camera to rendervirtual images to be presented to the eye of the user, said rendercamera having a position determined by said center of rotation.

Example 31: The display system of any of Examples 18-30, wherein saiddisplay is configured to project light into said user's eye to displayvirtual image content to the user's vision field at different amounts ofat least one of divergence and collimation and thus the displayedvirtual image content appears to originate from different depths atdifferent periods of time.

Example 32: A method of determining one or more parameters associatedwith an eye for rendering virtual image content in a display systemconfigured to project light to an eye of a user to display the virtualimage content in a vision field of said user, said eye having a cornea,said method comprising: with a plurality of eye tracking camerasconfigured to image the eye of the user and a plurality of lightemitters disposed with respect to said eye to form glints thereon,capturing a plurality of images of the eye of the user, said imagescomprising a plurality of glints; and obtaining an estimate of a centerof rotation of said eye based on the plurality of glints, whereinobtaining an estimate of the center of rotation of said eye comprises:determining a plurality of estimates of the center of corneal curvatureof the user's eye based on the plurality of glints; generating athree-dimensional surface from the plurality of estimates of the centerof the corneal curvature; and determining the estimate of the center ofrotation of the user's eye using the three-dimensional surface.

Example 33: The method of Example 32, wherein determining the pluralityof estimates of the corneal curvature of the user's eye comprises:determining a first vector directed toward the center of cornealcurvature based on the locations of at least a portion of the pluralityof light emitters and the location of a first camera of the plurality ofeye tracking cameras; determining a second vector directed toward thecenter of corneal curvature based on locations of at least a portion ofthe plurality of light emitters and the location of a second camera ofthe plurality of eye tracking cameras; and determining a region ofconvergence between the first vector and second vector to determine anestimate of the center of corneal curvature of the user's eye.

Example 34: The method of Example 33, wherein the first direction isdetermined by: defining a first plane that includes the first eyetracking camera, a location of a first glint reflection and a locationof the light emitter corresponding to said first glint reflection,defining a second plane that includes the first eye tracking camera, alocation of a second glint reflection and a location of the lightemitter corresponding to said second glint reflection; and determining aregion of convergence of the first plane and the second plane, theregion of convergence extending along the first direction.

Example 35: The method of Example 33, wherein the second direction isdetermined by: defining a third plane that includes the second eyetracking camera, the location of a third glint reflection, and alocation of the light emitter corresponding to said third glintreflection; defining a fourth plane that includes the second eyetracking camera, the location of a fourth glint reflection, and alocation of the light emitter corresponding to said fourth glintreflection; and determining a region of convergence of the third planeand the fourth plane, the region of convergence extending along thesecond direction.

Example 36: The method of any of Examples 32-35, wherein generating athree-dimensional surface from the plurality of estimates of the centerof the corneal curvature comprises fitting a surface to the plurality ofestimates of the center of the corneal curvature.

Example 37: The method of any of Examples 32-35, wherein generating athree-dimensional surface from the plurality of estimates of the centerof the corneal curvature comprises fitting a sphere to the plurality ofestimates of the center of the corneal curvature.

Example 38: The method of any of Examples 32-37, wherein determining theestimate of the center of rotation of the user's eye comprises:determining two or more vectors normal to the three-dimensional surface;and determining a region of convergence of the two or more vectorsnormal to the three-dimensional surface, wherein the region ofconvergence comprises the estimate of the center of rotation of theuser's eye.

Example 39: The method of any of Examples 32-38, wherein the pluralityof images of the user's eye comprise images associated with differentgaze directions of the user's eye.

Example 40: The method of any of Examples 32-39, further comprisingmapping the cornea of the user's eye using a gaze target.

Example 41: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising: a frame configured to be supported on ahead of the user; a head-mounted display disposed on the frame, saiddisplay configured to project light into said user's eye to displayvirtual image content; first and second eye tracking cameras configuredto image the user's eye; and processing electronics in communicationwith the display and the first and second eye tracking cameras, theprocessing electronics configured to: receive multiple pairs of capturedimages of the user's eye from the first and second eye tracking cameras;for pairs of images received from the first and second eye trackingcameras, respectively, obtain an estimate of a center of cornealcurvature of the user's eye based at least in part on the respectivepair of captured images; determine a three-dimensional surface based onthe estimated centers of corneal curvature of the user's eye obtainedbased on the multiple pairs of captured images of the user's eyereceived from the respective first and second eye tracking cameras; andidentify a center of curvature of the 3D surface to obtain an estimateof a center of rotation of the user's eye.

Example 42: The display system of Example 41, wherein said processingelectronics is configured to fit a three-dimensional surface to theestimated centers of corneal curvature of the user's eye obtained basedon the multiple pairs of captured images of the user's eye received fromthe respective first and second eye tracking cameras.

Example 43: The display system of Examples 41 or 42, wherein to obtainthe estimate of the center of corneal curvature of the user's eye basedat least in part on the respective pair of captured images, theprocessing electronics are configured to: determine a first vector alongwhich the center of corneal curvature of the user's eye is estimated tobe located based on a first image received from the first eye trackingcamera; determine a second vector along which the center of cornealcurvature of the user's eye is estimated to be located based on a secondimage received from the second eye tracking camera, the first and secondimages corresponding to one of said pairs of images; and identify aregion of convergence between paths extending in the direction of thefirst vector and the second vector to obtain an estimate of a center ofcorneal curvature of the user's eye.

Example 44: The display system of Example 43, further comprising: aplurality of light emitters configured to illuminate the user's eye toform glint reflections thereon, wherein to determine the first vectorbased on the first image of the pair of captured images, the processingelectronics are configured to: define a first plane that includes thefirst eye tracking camera, a location of a first glint reflection and alocation of the light emitter corresponding to said first glintreflection; define a second plane that includes the first eye trackingcamera, a location of a second glint reflection and a location of thelight emitter corresponding to said second glint reflection; andidentify a region of convergence of the first plane and the secondplane, the region of convergence extending along the direction of thefirst vector.

Example 45: The display system of Example 44, wherein to determine thesecond vector based on the second image in each pair of captured images,the processing electronics are configured to: define a third plane thatincludes the second eye tracking camera, the location of a third glintreflection, and a location of the light emitter corresponding to saidthird glint reflection; define a fourth plane that includes the secondeye tracking camera, the location of a fourth glint reflection, and alocation of the light emitter corresponding to said fourth glintreflection; and determine a region of convergence of the third plane andthe fourth plane, the region of convergence extending along thedirection of the second vector.

Example 46: The display system of any of Examples 41-45, wherein saidprocessing electronics is configured to use a render camera to rendervirtual images to be presented to the eye of the user, said rendercamera having a position determined by said center of rotation.

Example 47: The display system of any of Examples 41-46, wherein saiddisplay is configured to project light into said user's eye to displayvirtual image content to the user's vision field at different amounts ofat least one of divergence and collimation and thus the displayedvirtual image content appears to originate from different depths atdifferent periods of time.

Example 48: The display system of any of the examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 49: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising: a frame configured to be supported on ahead of the user; a head-mounted display disposed on the frame, saiddisplay configured to project light into said user's eye to displayvirtual image content to the user's vision field; an eye tracking cameraconfigured to image the user's eye; a plurality of light emitters; andprocessing electronics in communication with the display and the eyetracking camera, the processing electronics configured to: receiveimages of the user's eye captured by the eye tracking camera at a firstand second location, glint reflections of the different light emittersobservable in said images of the eye captured by the eye trackingcamera; and estimate a location of said center of corneal curvature ofthe user's eye based on the location of the glint reflections in saidimages produced by said eye tracking camera and based on the location ofthe eye tracking camera and the locations of the emitters that producedsaid respective glint reflections.

Example 50: The display system of Example 49, wherein said processingelectronics is configured to: based on the location of the glintreflections in one or more images produced by said eye tracking cameraand based on the first location of the eye tracking camera and thelocation of the emitters that produced said glint reflections, determinea first direction toward the center of corneal curvature of the user'seye; and based on the location of the glint reflections in one or moreimages produced by said eye tracking camera and based on the secondlocation of the eye tracking camera and the location of the emittersthat produced said glint reflections, determine a second directiontoward the center of corneal curvature of the user's eye.

Example 51: The display system of Example 50, wherein said processingelectronics is configured to determine the first direction by: defininga first plane that includes the first location of the eye trackingcamera, a location of a first glint reflection and a location of thelight emitter corresponding to said first glint reflection; defining asecond plane that includes the first location of the eye trackingcamera, a location of a second glint reflection and a location of thelight emitter corresponding to said second glint reflection; anddetermining a region of convergence of the first plane and the secondplane, the region of convergence extending along the first direction.

Example 52: The display system of Example 51, said processingelectronics are configured to determine the second direction by:defining a third plane that includes the second location of the eyetracking camera, the location of a third glint reflection, and alocation of the light emitter corresponding to said third glintreflection; defining a fourth plane that includes the second location ofthe eye tracking camera, the location of a fourth glint reflection, anda location of the light emitter corresponding to said fourth glintreflection; and determining a region of convergence of the third planeand the fourth plane, the region of convergence extending along thesecond direction.

Example 53: The display system of any of the examples above, whereinsaid processing electronics is configured to estimate a location of saidcenter of corneal curvature of the user's eye based on said first andsecond directions toward the center of the corneal curvature of theuser's eye.

Example 54: The display system of any of the examples above, whereinsaid processing electronics is configured to: determine said firstdirection along which the center of corneal curvature of the user's eyeis estimated to be located based on at least one first image receivedfrom the first location of the eye tracking camera; and determine saidsecond direction along which the center of corneal curvature of theuser's eye is estimated to be located based on at least one second imagereceived from the second location of the eye tracking camera, said firstand second directions converging toward a region.

Example 55: The display system of any of the examples above, whereinsaid processing electronics is configured to:

-   -   obtain an estimate of a center of corneal curvature of the        user's eye based on the convergence of the first and second        directions.

Example 56: The display system of any of the examples above, whereinsaid processing electronics is configured to estimate a location of saidcenter of corneal curvature of the user's eye by identifying a region ofconvergence of said first and second directions toward the center of thecorneal curvature of the user's eye.

Example 57: The display system of any of the examples above, whereinsaid processing electronics is configured to obtain an estimate of acenter of rotation of the user's eye based on multiple determinations ofthe center of corneal curvature of the user's eye for different eyeposes.

Example 58: The display system of any of the examples above, whereinsaid processing electronics is configured to determine a locus of pointscorresponding to estimates of the center of corneal curvature of theuser's eye for different eye poses.

Example 59: The display system of Example 58, wherein said processingelectronics is configured to obtain an estimate of a center of rotationof the user's eye based on said locus of points corresponding toestimates of the center of corneal curvature of the user's eye fordifferent eye poses.

Example 60: The display system of Examples 58 or 59, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye.

Example 61: The display system of Examples 58 or 59, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye by estimating a center of curvature of said surface.

Example 62: The display system of Examples 58 or 59, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye by determining a region where a plurality of normalsto said surface converge.

Example 63: The display system of any of Examples 60, 61, or 62, whereinsaid processing electronics is configured to fit said surface to saidlocus of points to obtain said surface.

Example 64: The display system of any of the examples above, whereinsaid processing electronics is configured to use a render camera torender virtual images to be presented to the eye of the user, saidrender camera having a position determined by said center of rotation.

Example 65: The display system of any of the examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content to the user's vision field at differentamounts of at least one of divergence and collimation and thus thedisplayed virtual image content appears to originate from differentdepths at different periods of time.

Example 66: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising: a frame configured to be supported on ahead of the user; a head-mounted display disposed on the frame, saiddisplay configured to project light into said user's eye to displayvirtual image content to the user's vision field; an eye tracking cameraconfigured to image the user's eye; a plurality of light emitters; andprocessing electronics in communication with the display and the eyetracking camera, the processing electronics configured to: receiveimages of the user's eye captured by the eye tracking camera at a firstcamera and second location, glint reflections of the different lightemitters observable in said images of the eye captured by the eyetracking camera; and estimate a location of said center of rotation ofthe user's eye based on the location of the glint reflections in saidimages produced by said eye tracking camera and based on said first andsecond location of the eye tracking camera and the locations of theemitters that produced said glint reflections for multiple eye poses.

Example 67: The system of example 66, wherein to obtain an estimate ofthe center of rotation of said eye, the processing electronics areconfigured to: determine a plurality of estimates of the center ofcorneal curvature of the user's eye based a plurality of glintreflections for multiple eye poses; and determine the estimate of thecenter of rotation of the user's eye based on the plurality of estimatesof the center of corneal curvature of the user's eye for said multipleeye poses.

Example 68: The system of example 67, wherein to determine saidplurality of estimates of the corneal curvature of the user's eye, theprocessing electronics are configured to: determine a first directiontoward the center of corneal curvature based on at least a respectivelocation of a portion of said plurality of emitters and a first locationof the eye tracking camera; determine a second direction toward thecenter of corneal curvature based on at least a respective location ofat least a portion of said plurality of emitters and a second locationof the eye tracking camera; and determine an estimate of the center ofcorneal curvature of the user's eye based on said the first and seconddirections.

Example 69: The display system of Example 68, wherein said processingelectronics is configured to determine the first direction by: defininga first plane that includes the first location of the eye trackingcamera, a location of a first glint reflection and a location of thelight emitter corresponding to said first glint reflection; defining asecond plane that includes the first location of the eye trackingcamera, a location of a second glint reflection and a location of thelight emitter corresponding to said second glint reflection; anddetermining a region of convergence of the first plane and the secondplane, the region of convergence extending along the first direction.

Example 70: The display system of Example 69, said processingelectronics are configured to determine the second direction by:defining a third plane that includes the second location of the eyetracking camera, the location of a third glint reflection, and alocation of the light emitter corresponding to said third glintreflection; defining a fourth plane that includes the second location ofthe eye tracking camera, the location of a fourth glint reflection, anda location of the light emitter corresponding to said fourth glintreflection; and determining a region of convergence of the third planeand the fourth plane, the region of convergence extending along thesecond direction.

Example 71: The system of any of Examples 68-70, wherein to determinesaid plurality of estimates of the corneal curvature of the user's eye,the processing electronics are configured to: determine a region ofconvergence between the first direction and second direction todetermine an estimate of the center of corneal curvature of the user'seye.

Example 72: The system of any of Examples 19-71, wherein to obtain anestimate of the center of rotation of said eye, the processingelectronics are configured to: generate a three-dimensional surfaceassociated with the plurality of estimates of the center of the cornealcurvature; and determine the estimate of the center of rotation of theuser's eye based on the three-dimensional surface.

Example 73: The system of example 72, wherein to generate athree-dimensional surface associated with the plurality of estimates ofthe center of the corneal curvature, the processing electronics areconfigured to fit a surface to the plurality of estimates of the centerof the corneal curvature.

Example 74: The system of example 73, wherein to generate athree-dimensional surface associated with the plurality of estimates ofthe center of the corneal curvature, the processing electronics areconfigured to fit a sphere to the plurality of estimates of the centerof the corneal curvature.

Example 75: The system of any of Examples 72-74, wherein to determinethe estimate of the center of rotation of the user's eye, the processingelectronics are configured to: determine two or more normals to thethree-dimensional surface; and determine a region of convergence of thetwo or more normals, wherein the region of convergence comprises theestimate of the center of rotation of the user's eye.

Example 76: The system of any of Examples 69-75, wherein the one or moreimages of the user's eye comprise one or more images associated withdifferent gaze vectors of the user's eye.

Example 77: The system of any of Examples 69-76, wherein the processingelectronics are configured to map the cornea of the user's eye using agaze target.

Example 78: The display system of any of Examples 66-77, wherein saidprocessing electronics is configured to use a render camera to rendervirtual images to be presented to the eye of the user, said rendercamera having a position determined by said center of rotation.

Example 79: The display system of any of Examples 66-78, wherein saiddisplay is configured to project light into said user's eye to displayvirtual image content to the user's vision field at different amounts ofat least one of divergence and collimation and thus the displayedvirtual image content appears to originate from different depths atdifferent periods of time.

Example 80: A method of determining one or more parameters associatedwith an eye for rendering virtual image content in a display systemconfigured to project light to an eye of a user to display the virtualimage content in a vision field of said user, said eye having a cornea,said method comprising: with an eye tracking camera configured to imagethe eye of the user and a plurality of light emitters disposed withrespect to said eye to form glints thereon, capturing a plurality ofimages of the eye of the user, said images comprising a plurality ofglints; and obtaining an estimate of a center of rotation of said eyebased on the plurality of glints, wherein obtaining an estimate of thecenter of rotation of said eye comprises: determining a plurality ofestimates of the center of corneal curvature of the user's eye based onthe plurality of glints; generating a three-dimensional surface from theplurality of estimates of the center of the corneal curvature; anddetermining the estimate of the center of rotation of the user's eyeusing the three-dimensional surface.

Example 81: The method of Example 80, wherein determining the pluralityof estimates of the corneal curvature of the user's eye comprises:determining a first vector directed toward the center of cornealcurvature based on the locations of at least a portion of the pluralityof light emitters and a first location of the eye tracking camera;determining a second vector directed toward the center of cornealcurvature based on locations of at least a portion of the plurality oflight emitters and a second location of the eye tracking camera; anddetermining a region of convergence between the first vector and secondvector to determine an estimate of the center of corneal curvature ofthe user's eye.

Example 82: The method of Example 81, wherein the first direction isdetermined by: defining a first plane that includes the first locationof the eye tracking camera, a location of a first glint reflection and alocation of the light emitter corresponding to said first glintreflection, defining a second plane that includes the first location ofthe eye tracking camera, a location of a second glint reflection and alocation of the light emitter corresponding to said second glintreflection; and determining a region of convergence of the first planeand the second plane, the region of convergence extending along thefirst direction.

Example 83: The method of Example 82, wherein the second direction isdetermined by: defining a third plane that includes the second locationof the eye tracking camera, the location of a third glint reflection,and a location of the light emitter corresponding to said third glintreflection; defining a fourth plane that includes the second location ofthe eye tracking camera, the location of a fourth glint reflection, anda location of the light emitter corresponding to said fourth glintreflection; and determining a region of convergence of the third planeand the fourth plane, the region of convergence extending along thesecond direction.

Example 84: The method of any of Examples 81-83, wherein generating athree-dimensional surface from the plurality of estimates of the centerof the corneal curvature comprises fitting a surface to the plurality ofestimates of the center of the corneal curvature.

Example 85: The method of any of Examples 81-83, wherein generating athree-dimensional surface from the plurality of estimates of the centerof the corneal curvature comprises fitting a sphere to the plurality ofestimates of the center of the corneal curvature.

Example 86: The method of any of Examples 81-85, wherein determining theestimate of the center of rotation of the user's eye comprises:determining two or more vectors normal to the three-dimensional surface;and determining a region of convergence of the two or more vectorsnormal to the three-dimensional surface, wherein the region ofconvergence comprises the estimate of the center of rotation of theuser's eye.

Example 87: The method of any of Examples 81-86, wherein the pluralityof images of the user's eye comprise images associated with differentgaze directions of the user's eye.

Example 88: The method of any of Examples 81-87, further comprisingmapping the cornea of the user's eye using a gaze target.

Example 89: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising: a frame configured to be supported on ahead of the user; a head-mounted display disposed on the frame, saiddisplay configured to project light into said user's eye to displayvirtual image content; an eye tracking camera configured to image theuser's eye; and processing electronics in communication with the displayand the eye tracking camera, the processing electronics configured to:receive multiple pairs of captured images of the user's eye from the eyetracking camera; for pairs of images received from the eye trackingcamera, respectively, obtain an estimate of a center of cornealcurvature of the user's eye based at least in part on the respectivepair of captured images; determine a three-dimensional surface based onthe estimated centers of corneal curvature of the user's eye obtainedbased on the multiple pairs of captured images of the user's eyereceived from the eye tracking camera; and identify a center ofcurvature of the 3D surface to obtain an estimate of a center ofrotation of the user's eye.

Example 90: The display system of Example 89, wherein said processingelectronics is configured to fit a three-dimensional surface to theestimated centers of corneal curvature of the user's eye obtained basedon the multiple pairs of captured images of the user's eye received fromthe eye tracking camera.

Example 91: The display system of Examples 89 or 90, wherein to obtainthe estimate of the center of corneal curvature of the user's eye basedat least in part on the respective pair of captured images, theprocessing electronics are configured to: determine a first vector alongwhich the center of corneal curvature of the user's eye is estimated tobe located based on a first image received from a first location of theeye tracking camera; determine a second vector along which the center ofcorneal curvature of the user's eye is estimated to be located based ona second image received from a second location of the eye trackingcamera, the first and second images corresponding to one of said pairsof images; and identify a region of convergence between paths extendingin the direction of the first vector and the second vector to obtain anestimate of a center of corneal curvature of the user's eye.

Example 92: The display system of Example 91, further comprising: aplurality of light emitters configured to illuminate the user's eye toform glint reflections thereon, wherein to determine the first vectorbased on the first image of the pair of captured images, the processingelectronics are configured to: define a first plane that includes thefirst location of the eye tracking camera, a location of a first glintreflection and a location of the light emitter corresponding to saidfirst glint reflection; define a second plane that includes the firstlocation of the eye tracking camera, a location of a second glintreflection and a location of the light emitter corresponding to saidsecond glint reflection; and identify a region of convergence of thefirst plane and the second plane, the region of convergence extendingalong the direction of the first vector.

Example 93: The display system of Example 92, wherein to determine thesecond vector based on the second image in each pair of captured images,the processing electronics are configured to: define a third plane thatincludes the second location of the eye tracking camera, the location ofa third glint reflection, and a location of the light emittercorresponding to said third glint reflection; define a fourth plane thatincludes the second location of the eye tracking camera, the location ofa fourth glint reflection, and a location of the light emittercorresponding to said fourth glint reflection; and determine a region ofconvergence of the third plane and the fourth plane, the region ofconvergence extending along the direction of the second vector.

Example 94: The display system of any of Examples 89-93, wherein saidprocessing electronics is configured to use a render camera to rendervirtual images to be presented to the eye of the user, said rendercamera having a position determined by said center of rotation.

Example 95: The display system of any of Examples 89-94, wherein saiddisplay is configured to project light into said user's eye to displayvirtual image content to the user's vision field at different amounts ofat least one of divergence and collimation and thus the displayedvirtual image content appears to originate from different depths atdifferent periods of time.

Example 96: The display system of any of the examples above, wherein atleast a portion of said display is transparent and disposed at alocation in front of the user's eye when the user wears saidhead-mounted display such that said transparent portion transmits lightfrom a portion of the environment in front of the user and saidhead-mounted display to the user's eye to provide a view of said portionof the environment in front of the user and said head-mounted display.

Example 97: A display system configured to project light to an eye of auser to display virtual image content in a vision field of said user,said display system comprising: a frame configured to be supported on ahead of the user; a head-mounted display disposed on the frame, saiddisplay configured to project light into said user's eye to displayvirtual image content to the user's vision field; at least one eyetracking camera configured to image the user's eye; a plurality of lightemitters; and processing electronics in communication with the displayand the eye tracking camera, the processing electronics configured to:receive images of the user's eye captured by the at least one eyetracking camera at a first and second location, glint reflections of thedifferent light emitters observable in said images of the eye capturedby the eye tracking camera; and estimate a location of said center ofcorneal curvature of the user's eye based on the location of the glintreflections in said images produced by said at least one eye trackingcamera and based on the location of the at least one eye tracking cameraand the locations of the emitters that produced said respective glintreflections.

Example 98: The display system of Example 97, wherein said processingelectronics is configured to: based on the location of the glintreflections in one or more images produced by said at least one eyetracking camera and based on the first location of the at least one eyetracking camera and the location of the emitters that produced saidglint reflections, determine a first direction toward the center ofcorneal curvature of the user's eye; and based on the location of theglint reflections in one or more images produced by said at least oneeye tracking camera and based on the second location of the at least oneeye tracking camera and the location of the emitters that produced saidglint reflections, determine a second direction toward the center ofcorneal curvature of the user's eye.

Example 99: The display system of Example 98, wherein said processingelectronics is configured to determine the first direction by: defininga first plane that includes the first location of the at least one eyetracking camera, a location of a first glint reflection and a locationof the light emitter corresponding to said first glint reflection;defining a second plane that includes the first location of the at leastone eye tracking camera, a location of a second glint reflection and alocation of the light emitter corresponding to said second glintreflection; and determining a region of convergence of the first planeand the second plane, the region of convergence extending along thefirst direction.

Example 100: The display system of Example 99, said processingelectronics are configured to determine the second direction by:defining a third plane that includes the second location of the at leastone eye tracking camera, the location of a third glint reflection, and alocation of the light emitter corresponding to said third glintreflection; defining a fourth plane that includes the second location ofthe at least one eye tracking camera, the location of a fourth glintreflection, and a location of the light emitter corresponding to saidfourth glint reflection; and determining a region of convergence of thethird plane and the fourth plane, the region of convergence extendingalong the second direction.

Example 101: The display system of any of the examples above, whereinsaid processing electronics is configured to estimate a location of saidcenter of corneal curvature of the user's eye based on said first andsecond directions toward the center of the corneal curvature of theuser's eye.

Example 102: The display system of any of the examples above, whereinsaid processing electronics is configured to: determine said firstdirection along which the center of corneal curvature of the user's eyeis estimated to be located based on at least one first image receivedfrom the first location of the at least one eye tracking camera; anddetermine said second direction along which the center of cornealcurvature of the user's eye is estimated to be located based on at leastone second image received from the second location of the at least oneeye tracking camera, said first and second directions converging towarda region.

Example 103: The display system of any of the examples above, whereinsaid processing electronics is configured to:

-   -   obtain an estimate of a center of corneal curvature of the        user's eye based on the convergence of the first and second        directions.

Example 104: The display system of any of the examples above, whereinsaid processing electronics is configured to estimate a location of saidcenter of corneal curvature of the user's eye by identifying a region ofconvergence of said first and second directions toward the center of thecorneal curvature of the user's eye.

Example 105: The display system of any of the examples above, whereinsaid processing electronics is configured to obtain an estimate of acenter of rotation of the user's eye based on multiple determinations ofthe center of corneal curvature of the user's eye for different eyeposes.

Example 106: The display system of any of the examples above, whereinsaid processing electronics is configured to determine a locus of pointscorresponding to estimates of the center of corneal curvature of theuser's eye for different eye poses.

Example 107: The display system of Example 106, wherein said processingelectronics is configured to obtain an estimate of a center of rotationof the user's eye based on said locus of points corresponding toestimates of the center of corneal curvature of the user's eye fordifferent eye poses.

Example 108: The display system of Examples 106 or 107, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye.

Example 109: The display system of Examples 106 or 107, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye by estimating a center of curvature of said surface.

Example 110: The display system of Examples 106 or 107, wherein saidprocessing electronics is configured to determine a surface based onsaid locus of points and to obtain an estimate of a center of rotationof the user's eye by determining a region where a plurality of normalsto said surface converge.

Example 111: The display system of any of Examples 108, 109, or 110,wherein said processing electronics is configured to fit said surface tosaid locus of points to obtain said surface.

Example 112: The display system of any of the examples above, whereinsaid processing electronics is configured to use a render camera torender virtual images to be presented to the eye of the user, saidrender camera having a position determined by said center of rotation.

Example 113: The display system of any of the examples above, whereinsaid display is configured to project light into said user's eye todisplay virtual image content to the user's vision field at differentamounts of at least one of divergence and collimation and thus thedisplayed virtual image content appears to originate from differentdepths at different periods of time.

Any of the above Examples can be combined. Additionally, any of theabove Examples can be integrated with a head mounted display. Inaddition, any of the above Examples can be implemented with a singledepth plane and/or with one or more variable depth planes (e.g., one ormore elements with variable focusing power that provide accommodationcues that vary over time).

Furthermore, apparatus and methods for determining a variety of values,parameters, etc., such as, but not limited to, anatomical, optical, andgeometric features, locations, and orientations, etc., are disclosedherein. Examples of such parameters include, for example, the center ofrotation of the eye, the center of curvature of the cornea, the centerof the pupil, the boundary of the pupil, the center of the iris, theboundary of the iris, the boundary of the limbus, the optical axis ofthe eye, the visual axis of the eye, the center of perspective, but arenot limited to these. Additionally, in some implementations, the centerof curvature of the cornea or the center of the cornea refers to thecenter of curvature of a portion of the cornea or the center ofcurvature of a spherical surface that coincides with a portion of thesurface of the cornea. For example, in some implementations, the centerof curvature of the cornea or the center of the cornea refers to thecenter of curvature of the cornea apex or the center of curvature of aspherical surface that coincides with a portion of the surface of thecorneal apex. Determinations of such values, parameters, etc., asrecited herein include estimations thereof and need not necessarilycoincide precisely with the actual values. For example, determinationsof the center of rotation of the eye, the center of curvature of thecornea, the center or boundary of the pupil or iris, the boundary of thelimbus, the optical axis of the eye, the visual axis of the eye, thecenter of perspective, etc., may be estimations, approximations, orvalues close to, but not the same as, the actual (e.g., anatomical,optical, or geometric) values or parameters. In some cases, for example,root mean square estimation techniques may be used to obtain estimatesof such values. As an example, certain techniques described hereinrelate to identifying a location or point at which rays or vectorsintersect. Such rays or vectors, however, may not intersect. In thisexample, the location or point may be estimated. For example, thelocation or point may be determined based on root mean square, or other,estimation techniques (e.g., the location or point may be estimated tobe close to or the closest to the rays or vectors). Other processes mayalso be used to estimate, approximate or otherwise provide a value thatmay not coincide with the actual value. Accordingly, the termdetermining and estimating, or determined and estimated, are usedinterchangeably herein. Reference to such determined values maytherefore include estimates, approximations, or values close to theactual value. Accordingly, reference to determining a parameter or valueabove, or elsewhere herein should not be limited precisely to the actualvalue but may include estimations, approximations or values closethereto.

1. A display system configured to project light to an eye of a user todisplay virtual image content in a vision field of said user, saiddisplay system comprising: a frame configured to be supported on a headof the user; a head-mounted display disposed on the frame, said displayconfigured to project light into said user's eye to display virtualimage content to the user's vision field; first and second eye trackingcameras configured to image the user's eye; a plurality of lightemitters; and processing electronics in communication with the displayand the first and second eye tracking cameras, the processingelectronics configured to: receive images of the user's eye captured bythe first and second eye tracking cameras, glint reflections of thedifferent light emitters observable in said images of the eye capturedby the first and second eye tracking cameras; and estimate a location ofa center of corneal curvature of the user's eye based on the location ofthe glint reflections in said images produced by both said first andsecond eye tracking camera and based on the location of both the firstand second eye tracking cameras and the locations of the emitters thatproduced said respective glint reflections.
 2. The display system ofclaim 1, wherein said processing electronics is configured to: based onthe location of the glint reflections in one or more images produced bysaid first eye tracking camera and based on the location of the firsteye tracking camera and the location of the emitters that produced saidglint reflections, determine a first direction toward the center ofcorneal curvature of the user's eye; and based on the location of theglint reflections in one or more images produced by said second eyetracking camera and based on the location of the second eye trackingcamera and the location of the emitters that produced said glintreflections, determine a second direction toward the center of cornealcurvature of the user's eye.
 3. The display system of claim 2, whereinsaid processing electronics is configured to determine the firstdirection by: defining a first plane that includes the first eyetracking camera, a location of a first glint reflection and a locationof the light emitter corresponding to said first glint reflection;defining a second plane that includes the first eye tracking camera, alocation of a second glint reflection and a location of the lightemitter corresponding to said second glint reflection; and determining aregion of convergence of the first plane and the second plane, theregion of convergence extending along the first direction.
 4. Thedisplay system of claim 3, said processing electronics are configured todetermine the second direction by: defining a third plane that includesthe second eye tracking camera, the location of a third glintreflection, and a location of the light emitter corresponding to saidthird glint reflection; defining a fourth plane that includes the secondeye tracking camera, the location of a fourth glint reflection, and alocation of the light emitter corresponding to said fourth glintreflection; and determining a region of convergence of the third planeand the fourth plane, the region of convergence extending along thesecond direction.
 5. The display system of claim 1, wherein saidprocessing electronics is configured to estimate a location of saidcenter of corneal curvature of the user's eye based on said first andsecond directions toward the center of the corneal curvature of theuser's eye.
 6. The display system of claim 1, wherein said processingelectronics is configured to: determine said first direction along whichthe center of corneal curvature of the user's eye is estimated to belocated based on at least one first image received from the first eyetracking camera; and determine said second direction along which thecenter of corneal curvature of the user's eye is estimated to be locatedbased on at least one second image received from the second eye trackingcamera, said first and second directions converging toward a region. 7.The display system of claim 1, wherein said processing electronics isconfigured to: obtain an estimate of a center of corneal curvature ofthe user's eye based on the convergence of the first and seconddirections.
 8. The display system of claim 1, wherein said processingelectronics is configured to estimate a location of said center ofcorneal curvature of the user's eye by identifying a region ofconvergence of said first and second directions toward the center of thecorneal curvature of the user's eye.
 9. The display system of claim 1,wherein said processing electronics is configured to obtain an estimateof a center of rotation of the user's eye based on multipledeterminations of the center of corneal curvature of the user's eye fordifferent eye poses.
 10. The display system of claim 1, wherein saidprocessing electronics is configured to determine a locus of pointscorresponding to estimates of the center of corneal curvature of theuser's eye for different eye poses.
 11. The display system of claim 10,wherein said processing electronics is configured to obtain an estimateof a center of rotation of the user's eye based on said locus of pointscorresponding to estimates of the center of corneal curvature of theuser's eye for different eye poses.
 12. The display system of claim 10,wherein said processing electronics is configured to determine a surfacebased on said locus of points and to obtain an estimate of a center ofrotation of the user's eye.
 13. The display system of claim 10, whereinsaid processing electronics is configured to determine a surface basedon said locus of points and to obtain an estimate of a center ofrotation of the user's eye by estimating a center of curvature of saidsurface.
 14. The display system of claim 10, wherein said processingelectronics is configured to determine a surface based on said locus ofpoints and to obtain an estimate of a center of rotation of the user'seye by determining a region where a plurality of normals to said surfaceconverge.
 15. The display system of claim 12, wherein said processingelectronics is configured to fit said surface to said locus of points toobtain said surface.
 16. The display system of claim 1, wherein saidprocessing electronics is configured to use a render camera to rendervirtual images to be presented to the eye of the user, said rendercamera having a position determined by said center of rotation.
 17. Thedisplay system of claim 1, wherein said display is configured to projectlight into said user's eye to display virtual image content to theuser's vision field such that the displayed virtual image contentappears to originate from different depths.
 18. The display system ofclaim 1, wherein said display is configured to project light into saiduser's eye to display virtual image content to the user's vision fieldat different amounts of at least one of divergence such that thedisplayed virtual image content appears to originate from differentdepths.
 19. The display system of claim 1, wherein said display isconfigured to project light into said user's eye that divergences and toproject light into said user's eye that is collimated to display virtualimage content to the user's vision field that appears to originate fromdifferent depths.